Author: Ho Yin Cheng
/
Created: February 13, 2018
I’ve forked and made minor pull requests in the past, but this past week (a full 3 days of analyzing/coding & 1 day of documentation), I made my first real fork of an existing project. It was quite the task as I had to learn the ins and outs of the existing codebase in order to make the large changes I wanted. Today, I’ll be talking about my entire thought process as I forked google-maps-services-js into react-native-google-maps-services.
I wanted to use the Google Places API, specifically Nearby Search, in the React Native app that I’ve been building and mentioning in my devlogs. The problem was that I was unable to find a satisfactory client library to use (something I’ll talk about later). So, I was left with the choice of either writing one from scratch or forking an existing repository and changing it to suit my needs. In the spirit of open source, I decided that forking was the way to go (it’s also a lot less work ;).
When people talk about Google Places API, there are actually several different parts to the API serving different functionality. This can get confusing when you search for information on how to use the API as you can get conflicting information as each has somewhat different functionality. It also gets very annoying when looking for client libraries as all of them use a generic name of “Google Places” but are actually built for only some of the APIs.
On top of that, the Places API is a subset of the APIs groups under the Google Maps API name. This can cause a lot of confusion when looking for information on how to use the different libraries.
Then on top of that, there are slightly different APIs exposed by the Android and iOS libraries. Both of which can be accessed by React Native if you choose to go the native module route.
Lastly, on top of that(!), there are different APIs exposed by the Places Library in Maps JavaScript API and others. There’s a lot to dig through and it’s made all the more confusing by the reuse of names. For my purposes, I’ll only dive into the web services APIs and a bit into the mobile APIs.
This was a multi-day effort of reading docs, diving into source code for poorly documented projects, and testing integration with React Native. This is what I discovered:
Official Client Libraries exist for Java, Python, Go, and Node.js.
If you want to try installing it into React Native, you can use the following libraries to get access to the Node.js core libraries (required since they aren’t part of React Native’s JavaScript engine; see also):
Installing the individual core libraries needed and using babel-plugin-rewrite-require to transform the requires that don’t have direct naming equivalents. And you’ll also have to install the q promise library for JavaScript as that is used in the code but no error is thrown when it isn’t found. Here’s an example of what you can put into your .babelrc
:
|
|
The core bug is this:
YellowBox.js:82 Possible Unhandled Promise Rejection (id: 0):
TypeError: Cannot read property 'getReader' of undefined
response.body
’s getReader()
method. However, there is no body
property of the response
. The question is why? There is a response
with data
. Just no body
(undefined
).fetch
implementation lags behind the official as can be seen by the dependencies:
"whatwg-fetch": "^1.0.0",
"node-fetch": "^1.3.3",
whatwg-fetch
lags behind the current release of whatwg-fetch (2.0.3).fetch
to be replaced by whatwg-fetch
by doing:
yarn add whatwg-fetch
whatwg-fetch
’s source code to not check for self.fetch
(fetch.js line 4)whatwg-fetch
. Previously, it didn’t because that check prevented whatwg-fetch
from replacing the native fetch
call. Which conflicts with how RN is checking for whatwg-fetch
(react-native/Libraries/Network/fetch.js and react-native/Libraries/Core/Devtools/symbolicateStackTrace.js). Yay for conflicting code bases!body
remained undefined
.fetch
response
’s body
.react-native-google-places-autocomplete
Nothing works.
Before going down the route of writing or forking a client library, it’s prudent to examine your original project needs to see if this is the best use of your time. And so, that’s what I did.
Some suggestions and comparisons to read:
These are the best of the best. Foursquare is the most attractive offering. Mapbox is great too as it would just be a part of their total offering (you can maybe even remove Google Maps completely). Agolia is a big name, but I’d rather go with Foursquare if I weren’t to try and use a completely Mapbox-only solution. Lastly, OSM is included because it’s the one self-hostable solution.
While these certainly do provide APIs that have the functionality I desire, none have an official React Native client library. Foursquare comes the closest but the libraries do not look very well maintained. As such, I felt that my time was better spent going with Google for now as it has the most robust looking code base for me to work off of.
If in the future I decide that I want to remove all Google services, I can probably reuse what I’ve learned from their codebase to write a more robust client library for Foursquare/Mapbox/etc.
With that out of the way, I made my decision to fork google-maps-services-js. I had to fork because:
makeUrlRequest
(#88 [#64(https://github.com/googlemaps/google-maps-services-js/issues/64)]). However, this wouldn’t work because we wouldn’t be able to remove the dependency on url
. It would also be more difficult to differentiate between put and get requests because of how the architecture of the code makes assumptions in makeApiCall
based on where it places value.makeUrlRequest
s even easier to create. This would allow swapping out axios much easier in the future. Passing around the config options in a sensible manner and making them available for assembly in the request.
requestUrl
in makeApiCall
, but it would remove the ability to swap out backends. I am also unsure of how reusing axios instances behave.Given that rationale, it was an easy decision to make. Furthermore, Github makes forking so easy that it seemed like a no brainer over trying to write from the ground up. Do keep in mind that while you work on your fork, the upstream repository may get some commits pulled in that you need to merge into your fork. I found these guides to be the most useful in understanding how to do this and how to write my commit messages:
These are the parts of the google-maps-services-js
source that use core Node.js modules:
google-maps-services-js/lib/index.js
util.deprecate()
to mark deprecated API methods.google-maps-services-js/lib/internal/cli.js
Process
to parse args.google-maps-services-js/lib/internal/make-api-call.js
Url
used to create the URL to send and to generate an encrypted signature.
formatRequestUrl()
- url.format
& url.parse
.Process
to grab environment variables.setTimeout
and clearTimeout
as default fallbacks.Buffer
to help generate the client secret.
formatRequestUrl()
and computeSignature()
.crypto
in conjunction with Buffer
to create an encoded signature.encodeURIComponent
to encode the signature.google-maps-services-js/lib/internal/make-url-request.js
https
uses to make a request.
axios
should come in.https.Agent
to set keep-alive
to true
.Url
’s parse
to parse the URL’s requestOptions
.Buffer
in conjunction with https
’s chunking ability to grab the response in pieces.https.request
is the root call that leads to the eventual body
is undefined
error.google-maps-services-js/lib/internal/task.js
Process.nextTick
to ensure the queued task will run in the next event loop.google-maps-services-js/lib/internal/wait.js
setTimeout()
for each task that is executed.clearTimeout()
for cancelling tasks.These will all have to be swapped out and replaced with a library like axios or superagent. A lot of the interface code, validation code, and even parts of the client side rate limiting should be reusable and remain untouched if done right.
Using my deep dive notes, I went through each task systematically to make sure that I didn’t miss anything that needed editing. Checklists for the win! You can follow along with my notes below as I explain the rationale behind all of my code edits.
google-maps-services-js/lib/internal/cli.js
Process
to parse args.
cli.js
and run.js
.google-maps-services-js/lib/internal/make-api-call.js
method
to the queryOptions to be later used in make-url-request
. This is either POST (explicit) or GET (implicit default) based on how they wrote their definitions.response.requestUrl
to equal requestUrlConfig
as we no longer create a url.
Url
used to create the URL to send and to generate an encrypted signature.
formatRequestUrl()
- url.format
& url.parse
.
https.request
.
params
object that I returned to be later used by makeUrlRequest
to create the axios instance.useClientId
is true
to create a encoded signature that is appended onto the end of your query string.
lib/apis
files to have baseURL
and relativeURL
params. This made it so I could use encodeURIComponent
to manually build the path + query string needed to do the encode.url.format
followed the WHATWG URL spec which looks to do quite a bit more than just encode. You can confirm this by seeing the extra cases that were discussed in this bug report: #1802. Then compare to how encodeURIComponent
works: [1] [2] [3] [4]fetch
because my experience with it in React Native has shown it to be very lacking. For example: #256 and that abort/cancel is still not finished (something needed by this codebase).Process
to grab environment variables.
process.env.NODE_ENV
which is shimmed in for compatibility purposes.setTimeout
and clearTimeout
as default fallbacks.
Buffer
to help generate the client secret.
formatRequestUrl()
and computeSignature()
.
buffer
is the best way to handle binary data in JavaScript. React Native appears to have a suboptimal implementation for anything related to this. See #10 in 10 Things you should know about React Native.crypto
in conjunction with Buffer
to create an encoded signature.
toString()
on the payload
and secret
for some reason as I was running into this bug: #32encodeURIComponent
to encode the signature.
google-maps-services-js/lib/internal/make-url-request.js
https
uses to make a request.
axios
should come in.lib/apis
files and the return from formatRequestUrl
.axios.create
in order to create an instance that will later send the request out.Buffer
point below.axios.cancel
with a generic cancellation message.https.Agent
to set keep-alive
to true
.
Connection
to keep-alive
which possibly does the same thing. However…httpAgent
and httpsAgent
params that take the very http/https
object that I removed. So this probably does not work as I expect.Url
’s parse
to parse the URL’s requestOptions
.
axios.create()
should build an instance with the correct default headers (and other params). So manually creating those params via Url
’s parse
shouldn’t be needed.url.parse
works at the low level, so my assumption above could be incorrect. A bug with the headers could have been introduced here. I may also need to use another library to have a keep-alive
agent as suggested by the answers here.Buffer
in conjunction with https
’s chunking ability to grab the response in pieces.
https.request
is the root call that leads to the eventual body
is undefined
error.
body
is undefined
in the http
response
due to the lack of streaming support). This is “fixed” by replacing https/http
with axios.google-maps-services-js/lib/internal/task.js
Process.nextTick
to ensure the queued task will run in the next event loop.
setImmediate
instead on React Native.google-maps-services-js/lib/internal/wait.js
setTimeout()
for each task that is executed.clearTimeout()
for cancelling tasks.
Showing 15 changed files with 92 additions and 160 deletions.
Explicit Dependencies Added:
Writing up some test code and putting the API through its paces in React Native showed that everything worked! Except geolocate
… why?
It turns out that geolocate
is no longer functional due to these outstanding issues/PR in axios: #723 #1342 #1121. Tracing the code all the way back, it looks like this bug was introduced in a combination of this commit and this commit that eventually changed the merge order of precedence like so:
So it was a consequence of not thinking through the history of changes and how axios recommends editing configs. The PR should fix this by changing the order to this:
For now, all I can do is wait to bump the axios version one the fix is merged and released.
Running the tests showed that some specs are failing. Meanwhile, all of Google’s spec tests pass.
My Spec Test Results
|
|
So why is this the case? My initial guess was that this had to do with using setImmediate
as the tests had to do with timing and task.js
. A quick edit of the original source to setImmediate
and a run later showed this to be partially the case:
|
|
So for comparison, that’s this difference which doesn’t align perfectly:
|
|
This is an oddity that is going to require a lot more investigation.
For the e2e tests, you’ll find that Google has not updated this library in a long time as the majority of their tests fail. When compared to my fork, you’ll actually get the same errors (+1 for removing the deprecated method):
|
|
I’m not sure how I’ll be tackling these errors as I have no passing baseline to compare against.
Overall, I would say that while work still needs to tbe done, the code is at least usable for my current purposes. If you would like to use this, be aware of the work I’ll be tackling in Future Tasks.
Now to make this fork usable by others without clashing with the original, I had to rename it on Github:
package.json
with the correct new information.Then publish it on npm with the new name:
And voila! You can now install and use this package with:
|
|
But a disclaimer: There are lots of outstanding issues that I need to fix before tagging this with a production ready release. So use this at your own risk!!
Forking an publishing projects under a new name isn’t all that simple though. There’s a lot of legal stuff and etiquette that I had to read up on first before I was sure that I should/can do this. IANAL but I’m pretty sure that the Apache 2.0 license that Google uses is ok with this given a bunch of caveats that I have to follow: [1] [2]
Also, remember that you’re building on the shoulders of giants when you fork and use open source work. Do your best to give back, but also be free to build on top of prior with with proper accreditation of course! [1] [2]
You can check the README for a list of immediate tasks that I’ll be tackling. However, in addition to those, I am also considering the following:
placesRadar
support so it can at least be used until it’s fully turned off.makeUrlRequest
.There’s more that can also be done like using it with React, Angular, etc. I’ve also run into some interesting articles on what you can do when implementing the Google Maps APIs. They’re good source for looking at features can be added after exposing the APIs themselves:
However, all of these extras are outside of my skillset and the current scope of the project. They’re all fun things to think about and hopefully I can tackle them in the future. On that note, I’ll end this post by saying that my first foray into open source has been super fun!
Copyright © 2011-2020 Ho Yin Cheng