Processing intensive workflows are higher in demand than ever, but the Mac lineup is missing a crucial piece to help pro customers of Apple.
AVM Fritz WiFi Mesh kam unerwartet ist aber ein Segen für den deutschen Markt
iOS is amazing but is lacking productivity basics
Update 2020-12-03 17:26
After searching a little and with help from Matt I was able to find the resource that stated that Github Actions is using MacStadium. The current version was wiped of that information but thanks to the Internet Archive I was able to find the actual information written out and disclosed by them.
If you haven’t yet I would recommend to donate to the Internet Archive, I just sent them $25,85
Following the Amazon AWS announcements that they will be joining the circle of few to offer Macs in their datacentres the topic around Mac hosting, macOS CI and which provider to pick has been widely discussed once again. The current overlap with the release of M1 based Macs and the remaining only half-to-poorly answered question of how to virtualize another OS on these Macs lead to various very interesting blog posts and public conversations on Twitter. Peter Steinberger sent me a DM asking me to fact check his article, and I figured I do this in public for others to read as well. I think he nailed pretty much everything but I wanted add a few things, starting by addressing his conclusion.
There’s no one-size fits-all solution when it comes to running macOS in the cloud. Both virtualization technology and bare metal are valid choices depending on organizational structure and requirements, but we hope this has given you a good overview of what’s possible.
He is absolutely correct here and missing critical information at the same time. There is no one-size fits-all solution to this, but there will always be one reason to choose virtualization: ephemeral builds.
These two magic words make anybody dealing with CI/CD infrastructure very excited for a very simple reason, which is predictability. It’s something our industry has seemingly been chasing for decades now and thanks to modern container technologies this goal is within reach or has been partially reached on the Linux side of things by the big CI-As-A-Service providers. Ephemeral builds means nothing more than always starting with the same environment. It provides a clean, predictable environment for your tests to run or your software to be packaged up and deployed. No lingering artifacts, crashed simulators or other things should exist which could disturb your fragile CI/CD pipeline. On the macOS side of things it’s our Linux on the Desktop. Next year will be the year, I am certain this time.
The way to achieve ephemeral builds “fairly” easily is by virtualizing the OS. A new VM is created every time a new build is started based on a copy of an environment that was setup specifically to your needs before. I know that this setup is possible with VMware vSphere as well as with KVM (MacStadium Orka). I helped maintain CircleCI’s VMware setup and build my own crazy little setup with KVM prior to the release of Orka, though I hadn’t used it for long. I think this would be possible with Veertu Anka as well but I have not tried it (yet).
A lot of MacStadium customers were interested in having this exact setup but had no idea how to get there or how to maintain it and ended up using virtualization without an ephemeral build system. They basically gave up before they even got there or a few months into it. The answer as to why mostly comes down to missing tooling, especially from Apple. You are playing Mac admin on extra hard mode and if that isn’t your day-to-day job it may be very hard to find the motivation to keep things running.
I had only started working on it when I left MacStadium and got very distracted since, but I am still absolutely convinced that ephemeral builds are possible on a bare metal system with APFS snapshots. Maybe I am wrong about it, but in theory this should be possible. If you know more I’d love to hear from you!
Virtualizing macOS will always present you with weird bugs since you are interacting with macOS in a way which it absolutely hates. The OS fundamentally wants to be operated by a human with a keyboard and a mouse attached to it, not by various automation tricks and scripts. Look at the state of macOS Automation and tell me that I am wrong. The guy that ran that entire thing for decades was fired (if I recall correctly) and left to join Omni and do cool things there. It perfectly shows that Apple’s leadership has no idea what to do with their best talent even if they already live in the Bay Area and have been working for them for years knowing all the ins and outs of how to navigate Apple internally.
Just don’t. You will be so much happier if you just don’t do that.
From Peter’s post:
I haven’t found a single writeup that takes price into consideration when discussing macOS virtualization. This is in some ways understandable, as most articles are from large companies, and engineers aren’t included in their price decisions. However, for smaller teams without venture capital, it’s an important metric.
Total cost is difficult to measure, since the promise of virtualization is less ongoing work, which should translate to reduced ongoing maintenance costs, and often employees are the most expensive cost factor.
Price mostly comes down to what your time is worth to you and/or whether or not you can find somebody capable and willing to maintain this infrastructure for you. Can you find somebody if that person leaves your company? What does it take to retain that talent and is there room for personal growth in this area?
To give you a real world example of my own, after giving a talk at Otto, which are a MacStadium customer, about exactly this topic I was approached by engineers from 4 different companies within 15 minutes asking if I could maintain their infrastructure for them as a contractor. There is a very good reason why this market is as underserved as it is.
Comparing Anka and Orka
The biggest (potential) upside to me with Anka is that you have the ability to turn all your VMs off and run a big build on bare metal macOS. This would require a sort of CI runner setup like Gitlab had for a long time or Buildkite and a sane CI tagging system. Maybe this would even be a way for some companies to run Android builds on the same host and get around nested virtualization, because after all I can’t stress enough how you should absolutely not try nested virtualization.
Fully Managed Services
TravisCI runs on machines at MacStadium and so does Github Actions. This was previously disclosed publicly by Microsoft when they ran their weird CI system but I would bet a lot of money that all of that has since been rolled into Github Actions after the acquisition.
There is chatter about the changed macOS EULA and how this relates to these services and my uneducated guess is this: They aren’t going to go anywhere. Companies will continue renting entire machines and will offer build minutes and Apple will keep looking the other way. Maybe Apple will ask for some changes but the fundamental services will not go away.
All of these are expensive for various reasons, the biggest is the manual labour involved in getting hosts online. AWS is not going to bend over backwards to get you anything and is not solving any of the hard problems with maintaining Mac build infrastructure.
What about the Mac Pro?
While I adore the Mac Pro and the solution Apple came up with to sorta-kinda-resurrect the Xserve, I think the underlying product is too expensive. If you don’t need the extra RAM or really know what you’re doing with a virtualized setup I would not recommend it. My idea of leveraging Anka comes into play again, but I have not tried this yet or compared any numbers.
The current Mac Pro is overly expensive and over engineered in ways that make it a very bad fit for datacentre use. I am assuming that the 19" rack mount version only exists for the music and video industry and that Apple thought little to none about it being used as a CI machine or living in a datacentre. In the words of a friend of mine at Apple about the Pro Display XDR: “This is not for you”.
This announcement from AWS left me a little confused. MacStadium has great compliance certifications so I doubt that most customers were waiting for AWS to join this space for these reasons. The prices are also outrageous and individual support probably a lot worse as compared to MacStadium. Unless you have to go with AWS for whatever reason, I see no point in signing up with AWS because as usual, they offering isn’t better than anything else by other vendors but they charge a 10x premium on it. I would recommend that you run macOS bare metal on Mac minis hosted by MacStadium if you can and use Buildkite to automatically kick of any builds. At that point you still haven’t solved any of actually hard problems of dealing with macOS and CI but I am sure that Peter will keep posting interesting things.
Like many other before me a few months ago I have noticed that I was incapable of focusing on longer-ish form text of any kind. I treated it similar to how I read my Twitter timeline, which is to say I mostly cross read until my mind finds something interesting it wants to focus on. Some days are worse than others but I got to the point where I wasn’t even able to read short-ish blog posts about topics anymore that genuinely interested me. In order to combat that behavior I actively looked for more long form text to read on a regular basis and once Dithering.fm was released I finally went ahead and subscribed to the Stratechery+Dithering bundle. It seems like a lot of money at first but it really isn’t if you break it down to a weekly or monthly cost and I do not regret it one bit. I think Ben’s analysis often gives me a perspective I wouldn’t be able to get otherwise. Since I started reading I routinely disagree with his point of view but that doesn’t mean that he is wrong or that I am right, it means that he has a different point of view and the exchange of thoughts and perspectives is why I keep reading. I think it’s good for me.
He offers many ways to read the Daily Update, including RSS or the typical E-Mail in your inbox and there is zero bullshit attached. I love it! Even the website is completely useable and legible no matter what device you’re using and you definitely do not have to jump into reader mode in Safari right away. It is a very good web experience which sadly is not common anymore.
Ben also offers the Daily Update as a Podcast but I force myself to not subscribe to it since I want to read his Daily Update. Me reading for myself is the important bit here. The only Daily Update that I listened to was his interview with Stewart Butterfield (@stewart), CEO of Slack. The back and forth in audio format was much better than the written text in my opinion.
Lets get this first bit out of the way: I bought a 3D printer. Yes I know which year this is, go and have your fun.
This is nothing new or world moving but since I enjoy having this as a tool to solve my own dumb little problems I figured I share how I solved a particular one: I printed hooks for my lightning cables.
If you’re an iOS developer you’re probably like me used to having more lightning cables and development phones flying around than you should. I has always bugged me to a certain degree to have these scrambled all over my desk until I found a little hook by Tesa (the glue and sticky stuff company) made to hang things with Power Strips (I think they’re called Command Strips in the US). The idea worked out well, the hook simply wasn’t big enough though for the amount of cables. With my newly acquired Prusa i3 MK3S I was now able to print this little hook and stick it to the back of my iMac which works perfectly. The hooks are within reach of your hand but out of sight.
In the back there you can also see my Sennheiser PXC 550 which I still enjoy thoroughly but the simple hook was a bit too small for it. I pre-printed the hook scaled to 175% and it is an absolute monstrosity compared to the original print but it also allowed me to use more Power Stripes and also wont damage the padding on the headphones now. 3D printers are pretty damn cool, who would have known.
Here you can see the small version compared ot the 175% version of the hook with AirPods Pro for reference (we were out of bananas. This is the internet after all)
And here is the final picture with the headphones on it. You can see that they sit nice and even on the hook.
If you’d like to have one as well there are printing services out there but they seem to be pretty pricy. Otherwise I would recommend the Prusa printers. It’s a kit that you have to assemble and it takes a good bit of time (I did it in about 8 hours over two days) but I am confident that most people can set it up. Their new Prusa MINI is pretty damn cheap as well.
Information has been, is and will always be the most valuable currency humans will ever be able to obtain. Hundreds of years ago spies were used to obtain information about an enemy, these days having access to information others do not, could still win a war or end up turning you into a millionaire.
Information is also a liability though. Many companies harvest as much data as possible from their users, or sometimes even from their paying customers and it has become industry standard to the point where most have become numb to this dystopian reality. Including those implementing these features who should know better. In the best case companies are most likely not setup and will never be setup to process this amount of data and in the worst case the people in charge at these companies are simply too stupid to even begin processing the smallest subset of the collected information. Too stupid in the sense that they keep collecting the information regardless of it’s value to them and because they probably also do not secure it properly. Lets hope that GDPR takes care of the latter problem.
(I think what follows now has started much much earlier but recent events brought this back to light once again.)
With recent Black Lives Matter protests the “Law Enforcement Community” has had the bright idea to tap right into the mighty data harvesting machinery enabled by all major companies. Why do the dirty work of tracking people illegally, storing the information for a lot of money and making employees sign their life away only to eventually ignore that and run away to Russia when you can simply buy the data when required. It has been long known that data brokers are willing to do business with just about anybody if they’re willing to pay the prices, and by doing business with the Police I’m sure they’ll get a little sticker saying that they have personally helped greatly in fighting terrorism or helped save the kids.
The only way to protect yourself is by simply not collecting data in the first place. Even having relevant information, being able to process it and derive conclusions which potentially leads to changing your product does not guarantee your product getting any better. You are what you measure, just ask Apple. All the collected customer sat data Tim loves so much, anonymised with differential privacy still leads them to an increasingly turned off pro market. If they can’t get it right, what makes you think that anybody actually knows what to do with all this information.
What other people decide to do is sadly out of reach of my control but I personally protect my privacy by using Pihole at home tied to 22.214.171.124 as my DNS provider, although I am not sure how much Cloudflare can be trusted either, and Guardian of course since I am working on both the app and the server infrastructure. With these in place I have recently noticed that advertisers are increasingly worse at reliably distinguishing between me and girlfriend. She has repeatedly received ads that were clearly targeted at me (stuff far out of the price range of a normal present or similar) and I have received more ads clearly not targeted at me or her at all. Seeing the mighty algorithm fail right in front of my eyes made me very happy. Ad tech and tracking algorithms have seemingly become so clever that the only way to escape them seem to be to not have information be collected at all in the first place. No data written leads to no context clues, leads to less click through, leads to less ad tech revenue which hopefully makes some of these companies cease to exist at some point.
I would encourage you to try one or both of these to protect yourself as well.
I think what many of us are talking about with the recent (5-10 years?) recession of Apple’s software quality has to do with the notion of Apples “something is better than nothing” approach to software. It is a state of mind so out of touch with what Apple prides itself in I am struggling to find the right words to describe it. The mDnsresponder rewrite for example was definitely not good enough and I believe all of would preferred not getting new features in an update than having a very critical part of a modern OS become so unreliable for no good reason.
The current trends in software development is a different one though. The race to the bottom forced by the App Store has spread onto all other platforms as well, but is felt on the Mac in particular (when looking at just Apple’s platforms) due to it’s open software ecosystem. Developers couldn’t even be bothered to implement the most basic functionality every user expects, which they’d get with no effort on their end by adopting native, or native-to-the-platform technologies.
Maybe this will get better, maybe this will end up getting worse with all platforms staying in existence and all of us being even more miserable or all of this may end up in us being like CGPGrey and ranting about Youtube sunsetting features due to lack of engangement on mobile, since Youtube failed to implement the feature to begin with (I can’t find the link, his rant is legendary). Developers don’t need yet another tool to make more mediocure software from other platforms (Apple’s own or others) into even more mediocure Mac apps. They need tooling and resources that allows them to share business logic etc. across the platforms and allows them to create good user interfaces. The latter part might be SwiftUI but I am not entirely sold on it just yet. The first part is definitely not Swift though, since it has only driven us further away from reliable, well crafted software.
A long time ago I posted that I was going move my E-Mail server away from OS X Server (RIP old friend) to something at that point still undecided in a project that I called XAPPLEPUSHSERVICE. I really did want to turn this into a series of multiple posts about how I switched, the options I considered etc. but I just didn’t have the time with my daily work schedule and having others depend on the E-Mail server being operational. I tested two options, opted for the second one and moved to a new operating system and E-Mail server setup over night while everybody that depended on it being online was asleep.
Just like all my other projects I set the server up to be as maintenance free as possible. Months and months or even years of uptime with no required update or me logging is always the target. Everything that can be automated in a reasonable fashion is automated and as long as I don’t get a message that something burns down I consider it to be online and doing what it’s supposed to be doing for me.
In the end I opted to format the SSD of my server to Ubuntu 18.04 Desktop Edition and installed mailcow. It’s a “mailserver suite”, which is essentially just a fancy term for various docker containers chained together to become something useful. A E-Mail Server.
Everything required to not go crazy operating a E-Mail server, including a web management UI comes with it and allows me to quickly take care of administrative tasks when needed. Users can also log into SoGo (which is written in Objective-C btw, the non-Apple reimplementation of Objetive-C) to get webmail access. It has yet to give me any trouble at all, apart from my own misconfigurations and is absolutely rock solid! I think you can run this setup on essentially any VPS and get yourself a great little E-Mail server which allows you to own your E-Mail data and not have to rely on the mood of any particular VP at some big company in Silicon Valley or Seattle. I do know that there are many other providers out there, I pay for and enjoy Fastmail a lot but they are the exception in an industry filled with big companies trying to lock users in with predatory behaviors.
I have been operating five domains with this one E-Mail server without any trouble for well over a year and can only recommend it. They also offer support packages in case you ever get yourself into some real trouble and hosted setups, meaning they setup and maintain the host but you own your data which in turn also supports further development of this “mailserver suite”.
One of the main downsides that I was prepared for but haven’t yet been able to solve again is the XAPPLEPUSHSERVICE part of this setup. I really miss not getting Push Notifications for incoming E-Mails and I really do not understand why Apple is not openly documenting the interaction, after all they do have to support with it with their own iCloud E-Mail service and have granted Fastmail an exception and allow them to send Push Notifications if you’re using their service through the builtin Mail.app and not through the Fastmail app that you can download from the App Store. Parts of the interaction are already documented by Apple hidden in their open source archive of OS X Server but some are not and maybe some of it has changed in iOS 11/12/13.