I spent this last Thanksgiving week at home with my family in the greater Boston area, which has been a wonderfully relaxing break from all the goings-on of my ordinary life in the Bay Area. Since I planned to get a little bit of work done for a client, I brought with me some development boards, including a Lattice MachXO3L eval kit. I had a productive Monday and Tuesday, getting quite a lot done, testing all the while in simulation, but by Friday, I was ready to test on hardware, and this posed a problem: the tool to program a bitstream onto the board runs on Windows and Linux, but I only had my Mac with me. I considered all the obvious solutions to this problem, weighed them each on their individual merits, and then discarded them all and chose the most complicated unreasonable approach possible.
There were a great many reasonable options. The truly obvious thing to do would be to install a Linux VM on my Mac, attach the device through the VM, install the programming software in the VM, and walk away happy. I was opposed to this idea for a handful of reasons: for one, my poor old Mac is running out of disk space; for another, my 16GB of RAM seemed already to be struggling under the load of approximately a billion WebViews from a bunch of tabs in Safari, a few tabs in Chrome, a handful of Slack instances in the Electron app, and a Signal instance in its own Electron app (grumble grumble, but the WebViews-taking-over-the-world rant is for another post...). I also didn't want to install more virtualization software. I realized that I did have some virtualization software on my Mac already, though! I have Docker for Mac installed, which I know runs a Linux machine in there somewhere! I can at least save the disk space cost. Docker uses VirtualBox or something inside, right?
Uh, wrong. docker-machine used VirtualBox, and indeed, there was a hacky solution to plumb USB devices to a Docker instance in VirtualBox, but... that feature's gone in Docker for Mac, since they switched to their own custom fork of xhyve. There's a feature request open for it in the Docker for Mac repo, but it's been largely ignored, with the only activity coming from a bot trying its hardest to autoclose the issue, and humans trying their best to say "no, this still matters to me". Ok, so Docker is right out.
But wait a second: I seem to remember back in the corners of my mind that Linux supports this "USB/IP" thing. And, will you look at that, there's already a multi-platform version of it: someone implemented a USB-over-IP host controller driver for Windows! So you could, in theory share a USB device that's physically attached to a Linux machine and give it to a Windows machine; or from a Linux machine, to a Linux machine. Presumably there's just some userspace tool or something that does the sharing, and a little bit of work could have that up and running on my Mac.
Aside: the astute will observe that the plot has already been lost, and the MachXO3 EVK is no longer the most important thing happening here.
USBIP is a curious architecture. When I went looking for documentation on it, and for information on how the protocol works, I got the classic "I believe I did, Bob!" answer: "for more information, go read our USENIX'05 paper". As best as I can tell, USBIP took their architectural model from NFS: a server "exports" a USB device, and a client imports it, presents a virtual USB host controller on the client machine, and then tunnels requests for the device over the network. This fundamental basis is sound, but the implementation's architecture was certainly an artifact of the times. In 2005, the world had not yet had the realization that perhaps not everything should be in the kernel; the NFS clientr lives in the kernel, of course, and so does the NFS server, so surely the USBIP client -- which has to expose virtual hardware -- should live in the kernel, and by extension, so should the USBIP server, right? And as was the architecture of USBIP: everything they built lived as kernel modules, and the userspace tools existed only to point the kernel in the right general direction.
This certainly makes the task of sharing a USB device from a Mac somewhat trickier. I theorized that it should be possible to implement a USBIP userspace server using libusb, and I found some header files that defined the network protocol for USBIP. It turns out that Wireshark supports USBIP, too, so I now had three implementations -- Wireshark, Windows, and Linux -- to compare notes between. So I figured that I'd give it a shot and see how far I got before my luck ran out.
I finally started typing code on Friday night. I nearly got caught in the "best of 2005" approach, too, when my old-hacker-mentality started visualizing ways to incrementally read packets in C without having decoded them yet, and began preparing to put together a select() loop wrapping around listen()ing sockets, already-accept()ed sockets, and also whatever file descripters libusb needed me to listen to, until I suddenly had the awakening that this is, in fact, 2018, and I can use, well, just about anything that isn't C. I figured the most likely language to have bindings for what I needed was Python, and if I had any hope of getting something quick and dirty working, that was what I was going to have to do. So I quickly closed my usbipd.c editor, and began googling for Python bindings for libusb.
Here's a quick side note: whenever you find yourself asking the question "are there Python bindings for foo?", you probably don't have to bother to ask. The answer is probably yes. There is the old saying that "if you ask ten Jews a question, you will get eleven answers"; the same seems to be true, in fact, that if you ask PyPI for one binding for a library, you will get a multitude of approaches. One of them will have exactly enough functions bound out to scratch some extremely narrow itch. One will be a direct swig mapping of function calls that crashes for 10% of them. One will be beautifully Pythonic, but against an old version of the library, and only complete right up to the part where coming up with a Pythonic interpretation of C functions suddenly becomes difficult. And one of them will be joyfully complete, if something of a messy approach, with the documentation for it being "well, the original library's documentation is pretty good, and so I didn't bother to write any, but I guess there are a few docstrings". You will mutter under your breath, and choose the latter; when you are done, you will be thankful for it anyway.
As it turns out, writing network code in Python these days is very nice, and asyncio basically does what you want; it seems that in Python 3, the batteries are finally included in that way with the async and await keywords. (Of course, it still makes me pine for the first-class coroutines in Lua that can yield at any point, but decorated functions are, at least, better than nothing!) In general, there were not a whole lot of surprises: python-libusb1 (not to be confused with python-usb) provided all the bindings I needed to asynchronously submit USB requests.
One thing that caught me out is that the USBIP protocol is, in fact, two protocols: a discovery and management protocol ("what devices are out there?", and "may I please claim device number seven?"), and an actual USB-over-IP protocol. These are multiplexed over the same socket link, and it seems to be only happpenstance that their packet formats can be intermixed; they use the same identifiers for sub-opcodes (i.e., USBIP_OP_IMPORT -- "I would like this device, please" -- and USBIP_RET_SUBMIT -- "here is that USB request response that you asked for" -- are both "opcode 0x0002"), but they have different headers. The way that you tell them apart is that opcodes in the actual USB-over-IP protocol are defined as being 32 bits, but they're all small, so the first 16 bits are always 0, but in the management protocol, the first 16 bits are the protocol version number (yes, embedded in each packet!), which are supposed to be non-zero. This was exciting and frustrating. Presumably 2005 was from before the era of design and code reviews in the Linux kernel, too.
Another surprise was that the Linux client is a little less stable than I had hoped for. Once I began sending USB data responses, I had the puzzling result that the remote end suddenly stopped talking to me. In fact, even when I went to export and re-import the device, the remote end *still* refused to talk to me; now it wouldn't even send an initial configuration request after the import packet! I took a look in dmesg, and noticed that my machine sitting in Mountain View was emitting alarming messages like task kworker/3:1:15678 blocked for more than 120 seconds. I tried to reload the kernel driver for the client; rmmod vhci-hcd hung forever. (It turns out that if you send an incomplete USBIP response packet -- that is to say, a header without enough data to follow it -- and then disconnect, then the other end's USB stack gets stuck until you reboot.) Kill count: 1.
I didn't want to reboot that server machine quite yet, since I was doing other stuff on it. So I did the only thing that made any sense: I installed a Linux VM on my Mac for development.
Saturday night, I finally had enough of the protocol implemented to talk to a FT2232 USB-serial chip -- the kind that was sitting on my Lattice board. I attached the USBIP device to my colo box in Sacramento with ssh -R port forwarding, and in doing so, made the first known USB-over-SSH connection, and certainly the first from a Mac. I launched the Lattice programming software, and 38 minutes of network latency round trips later, I successfully flashed a new firmware onto my MachXO3L board.
I packed up the box with the MachXO3L board and went to bed, since my flight was at 9am the next morning. I had saved at most 16 hours, flashing the board just an evening before I would arrive at home, where my Linux machine lived. I didn't use it on the plane.
pyusbip is available on GitHub. It has been lightly tested on Mac OS, connected to a Linux client, and currently supports only bulk endpoints and control packets to endpoint 0. I strongly suspect that I am one of quantity-single-digit people who intentionally insmod'ed a USBIP driver in 2018.
If you don't have a Dreamwidth account, but you want to get notified so you can read more things like this when I write them, I also have an e-mail list; I promise I'll only send mail for things that I write that go here on this blog. And, if this sort of skillset sounds like the kind of development work that you or your business needs, you can hire me through my consultancy, Accelerated Tech! (I promise to pick more efficient ways to get the job done for you, and don't worry -- of course my client didn't get billed for this little toy project...)