Using the Gargl Recorder Chrome Extension – A Step by Step Walkthrough

Hopefully by now you’ve learned a bit about what Gargl is all about, and why you’d use it. If not, take a look at my other blog post on Gargl first, Introducing Gargl: Create an API for any website, without writing a line of code.

Now that you know what Gargl is, lets talk about how using the Gargl Recorder Chrome Extension lets you easily record websites and generate Gargl templates. You can get the extension in the Chrome Web Store or add it to Chrome yourself from its source code here.


Gargl in action – Create an API for Yahoo in 3 minutes flat

After installing the Gargl Chrome extension, you can start using it by going into Chrome and hitting F12 to bring up the Chrome developer tools. You should notice on the far right a Gargl tab.

gargl1

 

Click on the Gargl tab. You can see in the Gargl tab that you can load an existing template file, or start from scratch on a new one. Let’s use Yahoo’s autocomplete and search “functions” to demonstrate how the Gargl recorder works. Navigate in the browser to the Yahoo homepage.

gargl2

 

Click ‘Start from Scratch’ and enter the module name and module description for the template file we’ll be creating. Since we only want to record the requests and responses against Yahoo, enter “yahoo” in the filter requests textbox.

gargl3

Click “Start Recording” to tell Gargl to start monitoring requests. Now let’s use the search bar on Yahoo’s site like we normally do. Don’t actually search, just type in the search bar to generate autocomplete results. As we type and autocomplete results roll in, we can see a request appears in Gargl for every letter we type. This is because Yahoo is getting updated autocomplete results from the server every time we type another letter.

gargl4

 

Looking in Gargl, you should see a lot of “search.yahoo.com/sugg” requests. Each of these represents a request / response that Gargl logged when Yahoo’s search page asked the server for autocomplete results.

gargl5

 

Now hit the “Stop Recording” button. Clicking on the details button for any of these requests will show us the details for the request. As you can see, it shows us the URL, method, and query string parameters of the request. For post requests it will also show the post data, of course.

gargl6

 

Since all of these requests are the same function (autocomplete, just with different ‘term to complete’ data), hit remove on all but the last “search.yaho.com/sugg” entry. For that last entry, enter “Autocomplete” for the name of the function.

gargl7

 

Hit the Edit button for the autocomplete function. As you can see there is a lot of data about the request / response of this function that was recorded and you can adjust. Enter a description for the autocomplete function. You could also change the request url here if you wanted to parameterize it (more on that below), but since this URL doesn’t contain any parameters that we care about, let’s leave it as is.

gargl8

 

Similarly, you could adjust or parameterize the request headers sent to the server for this function if you wanted. We don’t need to do that for this function though.

gargl9

 

You can also change the query string parameters sent in the request (or post data if the request is a post). As you can see, the “command” parameter had a value of “hello kitty is” when the request was sent. This is because this is what I was typing in the search box, which Yahoo wanted autocomplete suggestions from the server for.

gargl10

 

We could change the value of “command” to some other static value, like”spaghetti and “, to make the function always request autocomplete suggestions for “spaghetti and “, but lets instead create a function that takes in a parameter which becomes the value of the command query string parameter, so that the invoker of this function can specify what term to get autocomplete suggestions for. Setting the value of a key in the request to a parameter rather than a static value is done through a process called “parameterization.”

Parameterization is very easy to do —  just replace the current static value for the key with the name of the parameter you want to be a parameter to the function, surrounded by “@” signs. As you can see below, I want the autocomplete function of my Yahoo module to take a parameter named “term”, whose value becomes the value of the “command” query string parameter of the request that is made.

gargl11

 

Now hit “Save” to return to the page showing all your functions. If you now hit the “Details” button on the autocomplete function, you can see it contains our updated “@term@” parameter.

gargl12

Now let’s record our search function. Type a search query into Yahoo’s search bar (don’t hit Enter yet) and then click “Start Recording” in the Gargl tab.

gargl13

 

Click “Search” on the Yahoo page, or press Enter, to initiate the search request. You should now see a new request in the Gargl tab, with the URL “search.yahoo.com/search.” Name this request “Search”, it will become the Search function for our Yahoo module.

gargl14

 

Click the Edit button. You should see that the “p” query string parameter contains our search term.

gargl15

Change the search term to some parameterized input, like we did before for the autocomplete term. I chose for the function parameter to be called “query”.

gargl16

 

Unlike our autocomplete function’s response, which is formatted as JSON, our search function gets back HTML, since the page is displayed to the user. You can view the response of any request by clicking the “View Response” button on the function edit page. Clicking this for our search function gives us a big HTML file.

gargl17

 

Since we want our function to output some select data from the search results page, not a bunch of HTML, let’s figure out what the HTML elements that we care about in the returned HTML are. Let’s have our function return the titles of all the search results. Right click on a search result title on the Yahoo search results page, and go to “Inspect element.”

gargl18

 

As you can see from the HTML that gets highlighted in Chrome developer tools, the search result title is contained in an anchor tag, within an h3 tag. Clicking on other search result titles on this page reveals they fit the same pattern of an anchor tag within an h3 tag.

gargl19

 

Now let’s go back to our search function and make it grab this output instead of returning the entire HTML response it receives from the server. Click on the Gargl tab, then click Edit on the search function. Click “Add Response Field.” Here we can name every field we want the search function to output as part of the function’s output object. Enter a name for the field, and enter the CSS selector for anchor tags within an h3: “h3 a”.

gargl20

 

Hit “Test Selector” to have Gargl run the CSS selector you entered against the HTML response it recorded. As you can see below, it is correctly retrieving all the search result titles!

gargl21

gargl22

 

All done! Hit Save. We now have a Gargl module for Yahoo that contains functions for getting Yahoo search result titles for a term, and getting Yahoo autocomplete results for a term. In the Gargl tab, hit “Save As Gargl Template File” and then “Click to download.”

gargl23

 

This will download the Gargl template file (.gtf) for our module. I won’t go into the specifics of the schema of a Gargl template file here, but you can read up on it on GitHub.

gargl24

 

Great! Now we have our very own Gargl template! If we wanted to be a good Open Source Samaritan, we would add this Gargl template to the Gargl GitHub project here, so others could use a Gargl generator to turn it into a module in whatever programming language they want to integrate into Yahoo from. I’ve personally already uploaded this template to the Gargl templates directory, so we’ll skip that part.

Happy Gargling!

Interested in hearing about other side projects like this one? Subscribe to my blog and follow me on Twitter. I’ll let you know when I think of something fun.

9 thoughts on “Using the Gargl Recorder Chrome Extension – A Step by Step Walkthrough

  1. Nice job mate! I am trying to run my template under Linux (Ubuntu).
    Obviously I don’t have access to Power Shell, so I need to use node js.
    Could you please give me any advice how to do it?

    1. Hmm.. I am new in node js, but looks that some fixes in generated JS file will be needed (i.e. with jquery/jsdom requirements). I let you know when I solve this problems :-)

    2. After installing node, in bash console, go to the folder holding the Gargl node module.

      Make a new .js file, (abc.js) containing:

      var yourModule = require('./yourModuleName');
      yourModule.someFunctionInYourModule(someParameter1, someParameter2, function(err, response) {
      console.log(response);
      });

      Then in bash console:
      $ node abc.js

  2. Hi joe, it’s very nice job,

    But i have a problem,
    like your example, when i execute ‘node abc.js’, i have an error about jquery. it said ‘Cannot call method ‘ajax’ of undefined’, and i see in ‘mymodule.js’, we need to enable jquery. Where can i enable jquery ?

    Thank you =)

  3. Just spent hours trying to understand what I was doing wrong, when in fact the java generator is WIP and not functionnal at all as it is.
    At least the response fields are not supported for GET request.
    It would be nice to indicate it.

    1. Yes the Java generator does not support the response fields property yet. Can you open an issue on the github project to fix this, and also to indicate in the readme what is/is not supported per each generator language?

      1. Sorry I don’t get it, you are not supposed to be the developer of this project ?
        I think you know way better than me, what functions of this project you implemented or not.

  4. Hi,

    How can I get the content of a div with the CSS selector ?

    TEXT

    TEXT

    Is there any forum to get some help ?

    Thanks in advance,

Leave a Reply to bibiwood Cancel reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>