How to Upload a 4,000 Photo iCloud Photo Library into a 40,000 Photo macOS Photos Library to Free Up iCloud Drive Space

Today I discovered that I didn’t have enough iCloud storage space for a file I wanted to share.  In the past (apparently since sometime in 2015), I was able to find space here and there, but today, it appeared, I was out of options.  Actually, I don’t really use iCloud for much, storage-wise.  I upgraded my iCloud storage to the 50G plan, mainly just for backups for apps, email, and a few documents, but with all that extra space (coming from the free 5G plan), I had decided to turn on backups on my other iOS devices and… to turn on the iCloud Photo Library on my phone.  I didn’t really understand how it worked.  Apple always seems to be changing its photo synching services.  I had imagined it would be awfully nice if I didn’t have to physically connect my phone to my computer to upload my latest photos, because up until I activated the iCloud Photo Library, I would occasionally sync and select to remove photos from my phone after import.  So through some wishful thinking, my synching sessions got fewer and fewer and I noticed that it never seemed to free up much space.

Somewhere along the line, iPhoto gave way to the Photos app on macOS, and the “remove after import” checkbox went away and I figured they must have figured it out the way I’d previously imagined.  After each import, the photos would no longer be detected on the phone from the photos app, so I just hoped they were getting transferred.  I was confused that I still seemed to have photos on my phone – lots of them, but I still had space, so I just hoped everything was being magically handled.  Besides, I had other things to worry about, so I chose to ignore the problem and just hope that things were somehow magically and wirelessly getting to my computer.  I willfully put out of my mind and didn’t think about the fact that I had intentionally not turned on the iCloud Photo Library on my computer, because I knew I had 10 times as many photos there as I had on my phone.

Today however, I finally tallied the photos on iCloud, and it was over 20 gigs!  I tried synching my phone again.  I couldn’t tell where all the photos on my phone were being stored.  Why could I transfer some, and they would disappear, yet I had seemingly 3 years of photos on my phone.  Were the photos in the iCloud Photo Library also getting onto my computer?  It looked like some were in both places, but were they all?  I searched for a way to limit the space that photos would take up in iCloud.  I googled to find out whether I could safely delete photos from my phone(/iCloud?) and I got mixed messages.

The thing is, it’s not transparent where a particular photo is stored when I’m in the photos app and there’s no way to let my computer handle it the way it had in the past.  So I decided to recruit some help and called Apple.  I eventually got elevated to a Photos app expert (Jacob), and he explained things very well.  Here’s how I understood it:

  1. All the photos on my phone were being saved on iCloud, and only in iCloud.
  2. I had enabled “Optimize iPhone Storage” in the Photos Settings, which means that I only had compressed/smaller versions of the photos actually on my phone.  The full versions were in iCloud.
  3. When I synched my phone with my computer, I was only getting recent photos that hadn’t uploaded to iCloud yet.
  4. Those transferred photos weren’t removed from the phone, so they also ended up on iCloud (but anything that had uploaded to iCloud since the last time I synched, never made it onto my computer).
  5. The macOS Photos app does not synch the compressed/small versions of photos on my phone that were reduced after uploading to iCloud.

So I was faced with an issue.  Normally, one would just turn on the iCloud Photo Library on the computer and it would download all the high res photos, but doing that would be a 2-way synch and I had 10 times the amount of photos on my computer.  There wasn’t enough space in my iCloud drive to synch everything.  I could have bought more iCloud drive space, but I already have the photos on my laptop backed up on a NAS drive, so I’d prefer to just transfer the iCloud Photo Library to the computer.  So here’s how we did it…

  1. Quit the macOS Photos app.
  2. Start up the macOS Photos app while holding down the option key.
  3. Select to create a new photos library and name it something like “iCloud Photos”.
  4. Open System Preferences -> iCloud.
  5. Turn on Photos and click the options button.
  6. Turn on the iCloud Photo Library.
  7. In macOS Photos, click “Photos” at the top left.
  8. Scroll to the bottom and wait for the downloading message to change to “Updated Just Now”.
  9. Select all the photos and choose Export -> Export Unmodified Original…
  10. Save the photos in a new folder on the desktop.
  11. Quit macOS Photos.
  12. In System Preferences, turn off the iCloud Photo Library.
  13. Start up the macOS Photos app while holding down the option key.
  14. Select the original Photos Library.
  15. Select File -> Import…
  16. Select the folder you created on the desktop and click “Review for Import”.
  17. Wait for all the photos to be detected then select them and click “Import All New Items”.
  18. When the import is done, delete the folder you created on the desktop and the iCloud Photos Library that Photos created in your “Pictures” folder.
  19. [Proceed if you wish to remove the photos from iCloud, and thus your other devices…] Open System Preferences -> iCloud, then…
  20. Click the “manage” button at the bottom right.
  21. Click Photos, then click “Disable and Delete”

I never would have figured this out on my own and I imagine that many of you are just as bewildered as me, that is unless you started taking pictures after the iCloud Photos Library was created.

A couple things to note: 1. Photos you take in other apps (such as in the Messages app, whether they were taken and sent or received) are not in the iCloud Photos Library.  2. The macOS Photos app no longer detects duplicates, so importing photos that you previously synched (before they were uploaded to iCloud) may result in multiple duplicates.  There are apps available on the App Store for detecting and removing duplicates.

Advertisements

Work-around for Zombie Proximitask Reminders

I’ve posted about Proximitask before.  It’s no longer on the app store, despite it’s usefulness (especially when paired with RadBeacons which you can plug into any USB port).  I would post a Proximitask download link if any of them worked anymore, but if you still have it on your iPhone and use it all the time like I do for indoor location based reminders via iBeacons, you may have run into one annoying bug: you mark a reminder as done, but the reminder sometimes persists to pop up.  Here’s a way to permanently quiet those reminders… Continue reading

Using your university’s free Apple Developer account to eliminate the “Unknown Developer” Error in your 3rd-party-developed macOS App

If you work for a university and have developed a macOS app (as a part of your academic job) using 3rd party software (e.g. a Java app developed with Eclipse and built with Gradle), you’re probably familiar with telling your users how to deal with the annoying “Unidentified Developer” error they get when they double-click your app:

negcontrol.png

They get a window saying “<your app> can’t be opened because it is from an unidentified developer“.  The user is given no clear option on how to run the app except a vague reference to “security preferences”.

There are things the user can do to get around this error without you becoming an “identified developer”.  A user can perform a one-time bypass by right-clicking the app and selecting open or they can muck around with their Security & Privacy System Preferences settings.  They can even execute some commands in the Terminal to reveal a setting that has been hidden in the past few versions of macOS to always allow apps from anywhere.  But what would be nicer is to eliminate this error so that the user doesn’t have to do any of that.  Normally that involves giving Apple money to pay for membership in their developer program, but if you’re in academia, Apple has free such accounts for “Educational Institutions”.

Chances are, your university has already enrolled in this program and all you have to do is hook into that resource.  But be aware that universities have an image to uphold and that digital identity extends into the App Store, so they’re not likely to welcome any app to sit alongside their custom campus, reunions, and transit apps.  You can belay their fears about your app by noting that you intend to distribute your app outside of the App Store.  Users who download your app won’t even ever see the university’s name unless you added it to your app yourself.  All you need is a certificate that will allow you to code-sign your app so that Apple can know who you are.

It took me awhile to navigate this system.  I initially ran off on this tangent trying to create a free “education” Apple Developer account by digging up and supplying my University’s D-U-N-S number, official address, etc… and ran into a form that asked me to assert that I could commit the university to legally binding agreements, at which point I realized that I was on the wrong path.  Free educational institution Apple Developer accounts aren’t per person, faculty, or department.  They’re 1 per university.  I already had a couple of official university apps on my iPhone, so I knew there already existed an account, and made a number of calls before I got put in touch with the Apple Developer Account “holder”.

Those in charge of the Apple Developer Account here were only familiar with iOS app development, so they didn’t know how to advise me on accomplishing what I wanted to accomplish.  My sole goal was to eliminate the “Unidentified Developer” error.  No-one knew whether that meant I had to distribute my app on the app store or what, and they were of course concerned about the app showing up alongside all their campus-centric apps.  I managed to convince them that I wouldn’t be distributing on the app store and that I believed all I needed was a code-signing certificate.  Since no-one here knew how to do that, I called Apple again.

Apple really pushes their developers into Apple’s canned development resources (XCode) to do their development.  In fact, even their developer support team members tend not to know anything about code-signing and certificates, because it’s all handled in the background behind XCode.  I had numerous phone calls with Apple Developer Support and they proclaimed that it sounded like I was “speaking a different language”.  They weren’t even familiar with the “Unidentified Developer” error and immediately referred me to the makers of the 3rd party development apps I was using, implying it was a problem with their software.  I persisted however, because I had already tried code-signing using apple’s codesign command line utility and managed to get them to at least confirm that what I suspected was happening was happening.  And that is, that not any certificate will do… but I’m getting ahead of myself.

Here, I will walk you through, in detail, the steps I took to eliminate the error from my latest app (TreeView3)… Continue reading

Services to sum or subtract highlighted numbers and calculate equations

Years ago, I discovered the convenience of treating the Spotlight search as a calculator.  If you paste in an equation, the top result will be what the calculator app produces given the math you’ve typed or paste in.  I take a lot of notes in my work and frequently have a simple equation to calculate or numbers to sum or subtract.  It’s easy enough to copy and paste an equation into spotlight or paste a set of numbers (with some manual clean-up) to Excel or even search & replace with plus or minus signs & paste into spotlight, but you’d be surprised how much more efficient it feel to do this automatically.  You can even have it paste the result after the end of the highlight.

Combine this with the fact that many text editing apps allow you to select columns of text, and it becomes a trivial matter to do things with masses of numbers in any text and any app.

The sum & subtract automator services also conveniently strip out non-number characters, so you can highlight any numbers that have text in between and sum or subtract them.

Sum Service

Let’s start with the Sum service.  Here’s an example of how it works:

sum_demo.gif

As seen above, you simply:

  1. Select text with the numbers you want to sum
  2. Right-click the selection, and
  3. Select Services->Sum.

The numbers are parsed and added using Spotlight.  In the demo above, I simply tap the escape ket to exit spotlight after seeing the result.  But, you could select & copy the result to use it how you wish, directly from the spotlight result listing.

Subtract Service

The subtract service works the same way as the Sum Service above, but instead of adding all the numbers, it simply takes the first number and subtracts all the rest.  Both services ignore non-numbers and can handle decimals.  It even knows the difference between hyphens in words versus negative numbers.  However, they may have some trouble with non-standard hyphen characters and periods in full sentences.

Calculate (& Append Result) Service(s)

The Calculate Service (or “Calculate & Append Result”) will take any equation that Spotlight (i.e. the Calculator app) can handle and compute the result.  And if you know already that you will want to append the result after the highlighted text in the form of ” = answer”, you can alternatively select the “Calculate & Append Result” Service.  Here is a demo using the service:

calc_demo.gif

Installation

  1. Go to the github gist containing the Applescript code for each service
  2. Copy the code from one of the 3 files in the github gistcopygistcode.png
  3. Open Automator.appautomatordock.png
  4. Select service “Service”/gear icon from the dropdown sheet & click “Choose”selectservice.png
  5. Drag the “Run AppleScript” action into your workflowautomatorapplescript.gif
  6. Replace the purple code in the Run AppleScript action with the code you copied in step 1
  7. Save the workflow and name it however you would like it to appear in the services contextual menu (E.g. “Calculate & Append Result.workflow” – the extension will not appear in the menu)
  8. Repeat for the remaining 3 services.

The workflows/services will be saved automatically to your Library/Services directory in your home directory.  If you right-click the file name at the top of the window in Automator, you can select the Services folder to reveal it in the Finder.  You can then copy that file and send it to any other computer you would like to also have that service.

Have fun!

Disclaimers: These services are only intended as a quick and dirty solution to work in any context, & any app.  If you have a repeated common use-case, consider other solutions.  Note also that any application which reserves the arrow keys for some function when the shift key is held down, other than modifying the most recent selection, this service will fail.  Some applications, such as java applications, modify selections using shift-arrow navigation differently, depending on the direction of the mouse drag during text selection.  This can produce unexpected results.  A work-around for both such issues can be to use the strategy used for Terminal.app, but this would required modification to the code.  A few of the features in the script rely on some tricks such as statically set delays and command-line calls, necessary to either wait for an application to respond or to control the focus of various windows.  If your computer is very busy or has any configuration issues, the proper functioning of these services may be disrupted.  These services were developed and tested on macOS High Sierra, 10.13.4.  They may or may not work in other macOS versions.

Automator Services for finding coordinates in DNA/AA strings

Do you write code to analyze or modify DNA or proteins?  Do you do your work on a Mac?  If so, I have a few Automator Services, written in AppleScript, that you may find very handy:

  1. Get the sequence (& alignment) length of a selected nucleotide string
  2. Get the sequence length of any selected string (e.g. protein or quality string)
  3. Show where a coordinate is in any selected string (including white-spaces)
  4. Show where a coordinate is in a selected sequence (e.g. protein or quality string)
  5. Show where a nucleotide coordinate is in a selected sequence
  6. Show where an alignment coordinate is in a selected nucleotide sequence
  7. Get the reverse complement of a selected nucleotide sequence
  8. Guess the barcodes present in a FastQ file *NEW

With these, you can highlight a sequence anywhere in any application and either get the selection length or show where a supplied coordinate is in the selected sequence.

Each service, once installed, will show up in the contextual menu that shows up anytime you right-click any selected text, system-wide on your mac, under the services sub-menu, e.g.:

countntsservicemenu.png

Here are the full details of how to use each service and what it does:

1. Get the sequence (& alignment) length of a selected nucleotide string

Name: Count Nucleotides

countntsoutput.png

This service does a bit more than count nucleotides.  As seen in the example on the right, it reports the number of nucleotides in the selection (sequence length), the length of the selected alignment, the number of discrete & ambiguous nucleotides, and a breakdown of all case-insensitive sequence characters (including gaps and ambiguous nucleotides).

Spaces, tabs, newlines, carriage returns, numbers, or any other non-sequence characters are completely ignored, so you can select any sequence, even if it is formatted & displayed with coordinates.  Only the sequence found inside the highlighted text is considered.  The first selected sequence character is coordinate 1.

2. Get the sequence length of any selected string (e.g. protein or quality string)

Name: Count Sequence Characterscountnonwschars.png

This service counts every character selected except for spaces, tabs, newlines, and carriage returns.  It’s good for getting the length of selected unaligned protein sequences (with no formatted coordinates in the selection) or of quality strings.  Note, there is currently no service for aligned amino acid sequences, but if you would like such a service, let me know in the comments and I’ll whip one up.  I work mostly with DNA, thus I haven’t had much need for aligned protein coordinate determinations.

3. Show where a coordinate is in any selected string (including white-spaces)

Name: Select N Characters

This service works only on “solid sequence” (i.e. having no whitespaces, hard returns, or for that matter: any non-sequence characters).  See services 4-6 for sequence-specific functions.  The way this service shows where a coordinate is, is by changing the length of the selection.  The resulting last character of the selection after the length modification is the length supplied by you, the user.  For example, if you tell it to select 4 characters in this string you selected: ATGCCGTAG, the selection will end up as: ATGCCGTAG.

lengthprompt.png

There are 2 ways to supply the coordinate.  The default way is to grab the coordinate from the clipboard, so all you have to do is copy the number you want to use to set the selection length.  However, if the content of the clipboard is not a number, a popup window will appear to ask you to enter the desired selection length.

 

To use this service:

  1. [Optional] Copy a number/length indicating the amount of the sequence you want to select.
  2. Select any length of sequence from the start position (position 1)
  3. Right-click the selection and select Services -> Select N Characters
  4. [If you didn’t do step 1] Enter the length of sequence you want to select

This script alters the selection length of the selected text you right-clicked on in the window in which you clicked on the sequence, regardless of application.  However, if you are in Terminal.app, it displays the selection result in a popup window instead of in the Terminal itself.  This is because the modification of the selection length is accomplished by shift-arrow keystroke emulation and this is not a means by which you can modify a selection in the Terminal app.  This has 1 side effect.  Normally, if the entered length is longer than the selection, in any other app, that’s not a problem and the selection just expands, but in Terminal’s popup result window, all the service has access to is the selected text, so placeholder ‘N’s are appended to show the desired sequence length.

lengthresult-terminal.pngThere are other drawbacks to this Terminal selection-length work-around.  The font is not fixed-width and the width of the popup window is fixed at a fairly narrow size, thus large sequences cannot be displayed very well.

Since the selection modification happens via emulated keystrokes, it takes a little time for the final selection to be made, but you’ll see how fast it goes as you watch the selection being made.  Since it’s not instantaneous, the script will either adjust the selection from the end or select anew from the beginning for efficiency.

4-8. New services

Names: Select N Sequence Characters, Select N Nucleotides, Select N Alignment Characters, Reverse Complement, & Guess barcodes

These services operate just like Select N Characters, but take the character type into account.

Select N Sequence Characters doesn’t include whitespace characters such as spaces, tabs, and newlines in the calculation of a coordinate in a string.

Select N Nucleotides doesn’t include non-nucleotide characters such as white spaces, gap characters, numbers, etc. in the calculation of a coordinate in a string.

Select N Alignment Characters behaves just like Select N Nucleotides but includes gap characters in the calculation of a coordinate in a string as the one exception.

Guess Barcodes allows you to right-click on a file and find out what the likely barcodes are.  It takes a little bit to run and makes some common assumptions, but if you think the results are wrong, you are given the opportunity to tweak the parameters and run again.

Installation

  1. Go to the github gist containing the Applescript code for each service
  2. Copy the code from one of the 3 files in the github gistcopygistcode.png
  3. Open Automator.appautomatordock.png
  4. Select service “Service”/gear icon from the dropdown sheet & click “Choose”selectservice.png
  5. Drag the “Run AppleScript” action into your workflowautomatorapplescript.gif
  6. Replace the purple code in the Run AppleScript action with the code you copied in step 1
  7. Save the workflow and name it however you would like it to appear in the services contextual menu (E.g. “Count Non-Whitespace Characters.workflow” – the extension will not appear in the menu)
  8. Repeat for the remaining 6 services.

The workflows/services will be saved automatically to your Library/Services directory in your home directory.  If you right-click the file name at the top of the window in Automator, you can select the Services folder to reveal it in the Finder.  You can then copy that file and send it to any other computer you would like to also have that service.

Just try your new services out by right-clicking on selected text anywhere.selectncharsexample.gif

And as you can see from the example above, Select N Characters works on any text.

Have fun!

Disclaimers: These services are only intended as a quick and dirty solution to work in any context, & any app.  If you have a repeated common use-case, consider other solutions.  Note also that any application which reserves the arrow keys for some function when the shift key is held down, other than modifying the most recent selection, this service will fail.  Some applications, such as java applications, modify selections using shift-arrow navigation differently, depending on the direction of the mouse drag during text selection.  This can produce unexpected results.  A work-around for both such issues can be to use the strategy used for Terminal.app, but this would required modification to the code.  A few of the features in the script rely on some tricks such as statically set delays and command-line calls, necessary to either wait for an application to respond or to control the focus of various windows.  If your computer is very busy or has any configuration issues, the proper functioning of these services may be disrupted.  These services were developed and tested on macOS Sierra, 10.12.6.  They may or may not work in other macOS versions.

How to make Safari 10 auto-open specific file types

I’ve been annoyed for awhile now that Safari was auto-opening text files that I was trying to download in rapid succession via option-click on a series of links. The text documents kept popping up in front of my browser window and getting in the way.  Safari used to have preferences that you could set to tell it what to do with specific file types, but not anymore – at least not since Safari 10 on MacOS Sierra (10.12.x). The only options were to make Safari auto-open “safe” files or not auto-open any files at all. I was about to give up again.  I’ve tried to figure this out on multiple other occasions, but I’d finally had enough and decided to roll my own using Automator and some clever scripting… Continue reading

Turn On/Off Sighthound Security Cam Rules via IFTTT

We just moved, and in my last apartment, we lived on the second floor and I had my Foscam security camera [~$60] (which is controlled by the Basic version of Sighthound Video [$60]) trained on the only stairwell, which only captured people arriving or leaving. However in our new house, none of the doors are in locations where the camera only captures egress & ingress events.  There’s a bathroom right next to one door and the other is between the living room and the stairs.  Thus, anywhere I set up a security camera is bound to capture the regular activity inside the house. So I first experimented with setting the rules to make the cameras active during certain hours on the weekdays, which turned out pretty annoying, because my wife’s schedule is random.  What I needed was a way to turn the camera rules off & on based on when we arrive & leave the house, the same way I use WeMo to control the window unit air conditioners and the air filter.

I discovered that Sighthound has an IFTTT channel and an iOS app, but their IFTTT channel only has triggers upon motion detection and no “actions” that could be run using Life360 (the app I use to control the AC & filter – e.g. turn on when the first person arrives and off when the last leaves or vice versa).  So I emailed Sighthound to see if they had any tips for enabling and disabling camera rules.  They pointed me to one of their forums called Hacker’s Corner, where I got pointed in the right direction.

Here’s how to enable your security camera (rule) when the last person in your family leaves the house and disable the camera when the first person in your family arrives.

Continue reading