This is definately going to need AppleScript experience.

http://www.apple.com/applescript/safari/

open -a /Applications/Safari.app http://google.com

opens the default browser not Safari.

1) The java server isn't creating a proxy config when it starts up Safari

2) I don't know of a way to programatically change the proxy settings even if it was

3) Creating a proxy config by hand and setting the system to use it still doesn't work. It gets past the 404 error but then it hangs and eventually terms with a timeout.

server console output:
queryString = cmd=getNewBrowserSession&1=%2Asafari&2=http%3A%2F%2Flocalhost%3A3000
Launching Safari to visit http://localhost:3000/selenium-server/core/SeleneseRunner.html?sessionId=1159383080054 via /tmp/customProfileDir1159383080054/redirect_to_go_to_selenium.htm...
Got result: OK,1159383080054
queryString = cmd=open&1=%2Fsearch&sessionId=1159383080054
Got result:
queryString = cmd=testComplete&sessionId=1159383080054
Killing Safari...
Got result: OK

console output running test using Ruby client
rake test:selenium_search
(in /Users/michael/svnroot/rails-web-test)
ruby -Ilib:test test/unit/selenium_search_test.rb -n //
Loaded suite test/unit/selenium_search_test
Started
E
Finished in 31.895193 seconds.

1) Error:
test_search(ExampleTest):
SeleniumCommandError:
./lib/selenium.rb:153:in `do_command'
/opt/local/lib/ruby/1.8/timeout.rb:56:in `timeout'
/opt/local/lib/ruby/1.8/timeout.rb:76:in `timeout'
./lib/selenium.rb:139:in `do_command'
./lib/selenium.rb:403:in `open'
test/unit/selenium_search_test.rb:42:in `test_search'
test/unit/selenium_search_test.rb:41:in `test_search'

1 tests, 0 assertions, 0 failures, 1 errors
rake aborted!

http://developer.apple.com/documentation/Networking/Conceptual/SystemConfigFrameworks/index.html?http://developer.apple.com/documentation/Networking/Conceptual/SystemConfigFrameworks/SC_UnderstandSchema/chapter_4_section_2.html

MoreSCF:
http://developer.apple.com/samplecode/MoreSCF/index.html#//apple_ref/doc/uid/DTS10000702
"Mac OS X 10.1 introduced a new API, the System Configuration framework (SCF), that allows you to programmatically configure network settings. SCF provides a nice, abstract, extensible API for manipulating persistent preferences, however it is not a high-level API. Doing common operations, like creating a new set (known as a "location" in the user interface), requires you to combine a number of diverse APIs from I/O Kit, System Configuration, and Core Foundation. MoreSCF is an attempt to make this easier. It is modeled after the MoreNetworkSetup sample code for traditional Mac OS. It provides a library of code to do all of the common things that the user might do in the Network preferences panel."

Network settings – including the HTTP proxy – are stored on the file system in the following location:
/Library/Preferences/SystemConfiguration/preferences.plist

My test for whether I've configured a proxy or not is simply to try setting the proxy to "http://www.google.com". When I do that with the Network Preferences control panel, Safari fails to load any pages (including Google). I'm able to use the control panel to do this while Safari is running: check the box, Apply Now, refresh in Safari... homepage is broken. Uncheck the box, Apply Now, refresh in Safari... homepage works again.

Relevant threads on the macnetworkprog list:
http://lists.apple.com/archives/macnetworkprog/2006/Apr/msg00049.html
http://lists.apple.com/archives/macnetworkprog/2006/Apr/msg00061.html

1) ncutil seems to do the job, but you have to be root to use it to modify settings. (sudo is fine, of course.) Furthermore, if you want your settings to sink in prior to reboot, you have to restart configd. The only way to restart configd is to kill it and relaunch it as root. There's some documentation suggesting that configd is supposed to automatically notice changes to preferences.plist, but I can't reproduce that.

Even if we did require everybody to install ncutil, I'm not sure if I like the requirement that we run our software as root/sudo in order for us to automatically configure Safari's proxy.

2) as far as I can tell, scutil is a useful tool for reading settings, but its API for actually setting the settings doesn't appear to have any effect on anything, contrary to what Allan suggests in those threads.

scutil not only fails to update settings in the Network Preferences control panel, but it fails to modify Safari's proxy settings, regardless of whether Safari is running or not. This seems to contradict Allan's suggestion that scutil modifies the settings of running applications.

If scutil has some effect on some other program that isn't Safari, I can't see what it might be.

Here's what I'm doing:

open
lock
d.init
get Setup:/Network/Service/0/Proxies [unlike Nelson, I have only one]
d.add HTTPEnable 1
set Setup:/Network/Service/0/Proxies
unlock
show Setup:/Network/Service/0/Proxies
<dictionary> {
RTSPEnable : 0
ExcludeSimpleHostnames : 0
ProxyAutoDiscoveryEnable : 0
FTPPassive : 1
GopherEnable : 0
HTTPSEnable : 0
SOCKSEnable : 0
HTTPPort : 80
AppleProxyConfigurationSelected : 2
HTTPProxy : http://www.google.com
HTTPEnable : 1
}

The settings seem to persist when I quit/relaunch scutil, but they have no effect on Safari.

http://projects.info-pull.com/moab/MOAB-21-01-2007.html

This blog entry explains the more general case:

http://elliotth.blogspot.com/2006_03_01_elliotth_archive.html

"I've complained a few times about the number of setuid binaries on Mac OS. Various things that Linux manages to do via /proc, for example, Mac OS does via root-only API, and ships setuid binaries as helpers for those programs (such as Activity Monitor) that need the API nonetheless."

The writeconfig app and the NetCfgTool are both setuid. We may be able to use them (in their intended way, non-exploitatively) to set the network proxy on Safari without requiring a root password.

The System Preferences Control Panel absolutely uses NetCfgTool to do its dirty work (i.e. to set the network proxy as a non-root user). If you watch it in top, you can see that NetCfgTool isn't running when you start System Preferences, but it gets launched by System Preferences when you click on the "Network" button.

If you rename NetCfgTool so it's unavailable, you'll find that all network settings in the Network control panel are greyed out; you're unable to set them.

I had hoped that System Preferences was launching NetCfgTool with some command line arguments, but that's not apparently the way it works. (I tried renaming NetCfgTool to NetCfgToolx, putting in its place a simple bash script that logs command line arguments then calls NetCfgToolx. The log clearly showed no command line arguments.)

Apparently, System Preferences just leaves NetCfgTool running, and then somehow communicates with NetCfgTool using some kind of IPC mechanism. I don't know what that mechanism may be. Notably, a dude in this forum claims that you need to launch NetCfgTool in order to run scselect from the command line:
http://forums.macosxhints.com/archive/index.php/t-35301.html

I tried it and then tried running scutil from the command line, to see if I could get scutil to actually do anything, but it didn't work. I still have no evidence that scutil can set any setting whatsoever... scutil seems to be a totally useless app as far as I can tell.

My current best guess is that we would either:

1) require the end user to install ncutil and then ship a script that the end user should install setuid root, and then shell out to that script at runtime, or
2) if we could figure out how to communicate with NetCfgTool, we could launch NetCfgTool ourselves and then tell it to configure the network proxy without being root.

With that said, I'm going to go out on a limb and guess that we're not supposed to be doing that:
> strings NetCfgTool | grep service
No user serviceable parts included.

Anyway, it would be nice if we could monitor and track Apple Events, and see if that is the IPC mechanism being used in this case.

$ /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support/networksetup -help

This has options like:
Usage: networksetup -setwebproxy <networkservice> <domain> <port number> <authenticated> <username> <password>
Usage: networksetup -setwebproxystate <networkservice> <on off>

also has setsecurewebproxy[state]

<networkservice> is one of (on my box):
$ networksetup -listallnetworkservices
An asterisk denotes that a network service is disabled.
Bluetooth
Built-in Ethernet
Built-in FireWire
AirPort
VPN (PPTP)

So for me, I was able to invoke these commands, and then verified the settings in the Network Sys Pref panel, and also verified that Safari either worked (with proxystate to off) or did not
(when proxy was localhost:4444 w/ no Se running).

I think I might start to work on a patch... I am rather unfamiliar with your codebase however, where would this sort of command-line stuff go?

We are already using some process spawning code in the browser launcher code. Look at the users of the AsyncExecute module in org.openqa.selenium.server.browserlaunchers (though of course I don't think you'll want to be asynchronous in this case; AsyncExecute is a child of Execute, which may well be the way to go for you).

But beware: on my 10.4.5 machine, you won't find "networksetup" in /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support/. Instead, on my box it appears to be called "networksetup-panther"! That may make things a bit trickier... maybe look for an executable whose name starts with "networksetup"?

Oh, and to address your earlier question, I would compare/contrast OS X "networksetup" support with our implementation of Windows registry support for IE; many of the problems that apply there would apply here also. We need to shell out to call reg.exe on Windows to do our work; we need to use it to back up the settings before we modify them, and we need to safely restore those settings when we're cleaning up. If we crash before we can clean up, we need to be clever enough not to overwrite the old backup, and to restore the old backup safely if/when we finally do get a chance to clean up.

This may be a lot to bite off at once... Even a starter patch would be much appreciated!

-Tom

PS> Can someone else with a Mac who reads this verify that you really really don't need to be root to do this? I read somewhere that changing the params really does require sudo access... I am thinking that might be true in the event that the users have the 'Click the Lock to make changes.' set in Network Preferences. Which, if that is the case, then they would have to unlock that to manually change it anyway, so I'm okay proceeding...

Thanks again, Thomas, for looking into this.

-Dan

-Tom

This is as much of a test of the selenium trunk as the patch. I'm not an OSX person, nor a Selenium expert or maven expert. The latest version of networksetup on my system is networksetup-panther... which seems to add to the versioning annoyances above. I needed to make a symlink from networksetup to networksetup-panther.

0. got trunk
1. Patched selenium-coreless/**/SafariCustomProfileLauncher.java (r1854) with patch above, had to manually integrate some portions
2. Rebuilt and installed the 9.2-SNAPSHOT into the maven repository, updated my pom to point to 9.2-SNAPSHOT
3. ran the integration tests with sudo (otherwise there would be a permissions problem copying /Library/Preferences/SystemConfiguration/preferences.plist.old)

TEST 1: Safari would do the same thing it did under the release version, that is startup and display a 404
TEST 2: Firefox would start performing tests (abeit slowly... until halting after the ThreadPool complained)

All tests would halt after the following message, not related to the patch. I'm looking into these.
[INFO] 10:27:19,586 INFO [org.mortbay.http.SocketListener] LOW ON THREADS ((512-511+0)<2) on SocketListener0@0.0.0.0:4444
[INFO] 10:28:19,630 WARN [org.mortbay.util.ThreadPool] No thread for Socket[addr=/127.0.0.1,port=57432,localport=4444]

I independantly discovered the networksetup trick to set the proxy and implemented it in a shell script that I am wrapping selenium rc calls in. However, the above error is very consistent. I tried running the server with "-jettyThreads 1024" which caused out of memory errors, so I increased my heap size to 256M and voila, I got the same error message but that contained
LOW ON THREADS ((1024-1023+0)<2) instead. 1024 is the maximum set in SeleniumServer.java and I don't think increasing it will actually help. I tried adding the -avoidProxy flag which made no difference.

I also thought the problem might be in my jre but I have tried with both 1.5.0 and 1.6.0-dp and neither made any difference save for modifying the error message slightly.

I'm on OS X 10.4.10, kernel version Darwin 8.10.0 PPC

I've been testing with the latest selenium-server-0.9.2 snapshot as of this morning, a 0.9.1 snapshot from July and the 0.9.0 release.

The RC server has to be started with the following options (assuming you want pimode)

java -jar -Dhttp.proxyHost='' -Dhttp.proxyPort='' selenium-server-0.9.2-SNAPSHOT-standalone.jar -proxyInjectionMode -forcedBrowserMode '*safari'"

The reason for the proxyHost and proxyPort is that once you've started the system-wide proxy with networksetup, Java will try to route even your Selenium traffic through it, back through itself, and you'll run out of threads.

If you set the proxyHost and proxyPort to '', then Java will not use a proxy, and it will work just fine(ish).

/System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support/networksetup -setwebproxy "Airport" localhost 4444
/System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support/networksetup -setwebproxystate "Airport" on

java -jar -Dhttp.proxyHost="" -Dhttp.proxyPort="" "selenium-server-0.9.2.jar" -userExtensions "user-extensions.js" -htmlSuite "*safari" "http://...." "TestSuite.html" "ResultsFile.html"

/System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support/networksetup -setwebproxystate "Airport" off

This is now working for me.

$ /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support/networksetup -setwebproxy "Built-in Ethernet" localhost 4444
cp: /Library/Preferences/SystemConfiguration/preferences.plist.old: Permission denied

Here's some more info on that file:
$ ls -la /Library/Preferences/SystemConfiguration/preferences.plist.old
rw-rr- 1 root admin 14591 Sep 24 12:57 preferences.plist.old

Tested on a fully patched Mac Pro desktop and MacBook Pro laptop.

I'm starting to think a long term solution here will be AppleScript based. (At least until Apple requires sudo access to modify the network preferences GUI.)

Don't forget that it's very easy for people to think that they've configured their proxy correctly when, in fact, they're confused about which setting they really need to set and where to look to see if their commands had the expected results. For example, people think they're using Built-In Ethernet when in fact they're using AirPort, or vice versa.

IMO, AppleScript is not the way to go here. AppleScript requires automating the UI of the Control Panel, including parsing the Control Panel GUI to enumerate the available network services, back up their settings, and then modify them safely. I can't believe that's the right answer here, relative to some other command-line based solution. Even writing our own proxy configuration program in C would likely be a better way to go.

Regarding the AppleScript vs C comment... well no comment, since networksetup is in fact working for me. But in general, you're right-- scripting the GUI is always the option of last resort and usually the worst option, unless... it's your only option. Perhaps, that should be the motto of the Selenium project.

0) scutil really shouldn't be used for writing the proxy settings, but it is an effective way of reading the current effective settings out of the dynamic store. For example, you can determine the current "primary" interface (e.g. "en0") as well as the primary network service name (used for networksetup) by using "show State:/Network/Global/IPv4".

1) It may seem as if it's possible to configure network settings as non-root without using a password, but in fact to get it to work you need to be an "admin user." Even the setuid root tool NetCfgTool requires "admin" privs. The wrinkle is that most Mac systems have a single user who has these privs and the "lock" is generally open. If you go to Network Prefs and close the lock then you should get a prompt when attempting to use networksetup.

2) As we already knew, there are many varieties of "networksetup" out there in the wild, at least some of which are unable to configure certain proxy settings. In particular, some versions of networksetup have commands to "-getautoproxyurl" "-setautoproxyurl" and "-setautoproxystate", whereas other versions of networksetup don't have this feature. (In particular, on the Intel 10.4.10 iMac I've got here at work, it does NOT have that feature.)

3) NetCfgTool will be "essentially gone" post-Tiger.

4) All this made me want to research ncutil again. I found that ncutil 3.1 doesn't immediately affect configd unless you "nudge" it with the "apply-changes" command. You can also pass the "--apply-on-exit" option to do the same thing. These commands will affect the currently running Safari just the way you'd expect:

sudo ncutil --apply-on-exit setprop "/Automatic/Built-in Ethernet/Proxies" http-proxy-enable 0
sudo ncutil --apply-on-exit setprop "/Automatic/Built-in Ethernet/Proxies" http-proxy-enable 1

ncutil is easily able to modify all of the proxy settings, including the auto proxy url.

5) Using a VPN can really screw up the whole thing. We use the Cisco VPN for OS X here on the Mac at work, and I found that the VPN appears to "capture" the proxy settings when the VPN connects. Once the VPN is on/connected, nothing you can do will change the proxy settings, even when you change them manually using the Network Control Panel.

For example, you can set the proxy manually to www.google.com:80, then start Safari, and find that every site looks like Google. Then start the Cisco VPN and finally turn off the HTTP proxy. You'll find that every site will still look like Google until you disconnect the VPN, at which point your network will resume paying attention to the proxy settings you configured in the control panel.

IMO, what we should do on top of this is write a shell script that wraps around ncutil and uses it to write out HTTP proxy settings. We should make it simple and write it in a highly defensive style, and advocate to our users that they install this script on their path with setuid root.

Then, for our part, we should look to see if the script is available (perhaps adding [ulp] yet another command line argument) as well as ncutil. If we find everything we need, we use ncutil for everything. (ncutil is BSD code, and so I suppose we could just provide it ourselves, but I think installing it requires root access, and creating that setuid root script would DEFINITELY require root access; ideally, I'd like not to have to run the whole Selenium Server as root just to configure the proxy settings.)

If ncutil is not available, we should fall back to using scutil + networksetup. (We should query networksetup to see if it supports configuring PAC files and use that functionality if it's available.) If networksetup isn't available, then we should fail with a link to a web page explaining our situation, like the one we've already got for Windows reg.exe issues: http://wiki.openqa.org/display/SRC/Windows+Registry+Support

As you can tell, there's a LOT of special cases for this code; I made sure that I unit tested the hell out of the code I've already checked in, and there's still way more code not yet written that will need a similar treatment.