Working with the Search Console API | Search Off the Record

[Music]
hello and welcome to another episode of
search off the record a podcast coming
to you from the Google search team
discussing all things search and having
some fun along the way my name is Martin
and I'm joined today by John from the
search relations team of which I'm also
part of hi John hi Martin also with us
in the studio is Miis Mii hi hey hey
John hey Martin thanks for having me
would you be so kind to introduce
yourself to our listeners sure so I'm an
SEO Enthusiast and entrepreneur based in
Romania and I started tinkering with
SEO um almost 20 years ago about 2006
something like that oh wow uh yeah I was
uh in back in my University days uh as a
student and
uh I really like figuring out how things
work and SEO was this interesting
mysterious thing that uh a lot of people
are trying to figure out so I did what
most other people were doing back then
is try to create a few websites and try
to get them to rank basically um so that
slowly morphed into a kind of a
freelancing job where I started to do
SEO for other people's websites and and
um after finishing my master's degree in
Bucharest I decided to open my own
agency and that I think that was around
2011 2012 wow so you've been around for
a while huh yeah I basically just tried
to transform my hobby into a legitimate
business and yeah ever since then my
passion for SEO constantly grew uh so I
created a few SEO tools along the way
and also tried to get myself to involved
as much as possible into the SEO
Community First by joining a lot of
John's SEO office hours
and and then becoming a uh product
expert although I think it was a top
contributor back then uh on the Google
uh official search Central forums cool
and that's that's why you're here in the
office today because uh we're doing a
product expert Summit in Zurich at the
moment and we thought it would be a
great idea to maybe invite someone from
the product expert group to join us for
the podcast yes so I don't know if
everyone out there listening to the
podcast knows the help Community or the
forums as we call them um would you like
to introduce them a little bit like how
how does the community work how does the
Forum work from your perspective like
who goes there what goes on
there how can I as a non SEO navigate
these places how does that work sure so
I can tell you why I join the forums in
the first place uh first of all just as
with uh John's SEO office hours uh I
think the forums are a good way to kind
of bridge the gap between the official
documentation which is kind of catered
toward you know broad uh it has kind of
a broad appeal um whereas the office
hours and the forums kind of cater more
towards specific use cases and issues
that people might have um it's also the
case that I I know that um googlers
can't really come up with their own
personal opinions so uh I think it's a
good um chance for us PES uh which are
non-googlers
to use our own experience and come up
with suggestions that are our own and be
able to help people along the way that's
pretty cool um it's also a learning
opportunity for me because I get to see
some issues that I might have not
experienced myself before and uh it's
just a joy to help out other people yeah
I really like the forums as well exactly
for the reason that you said um the
documentation has to be like Brad
because we don't know what the context
is what people are coming to the
documentation for is they like are they
trying to learn about the topic are they
trying to fix a specific problem what
kind of environment are they in and in
the Forum you can have these kind of
discussions like hi my WordPress site
has this symptom what could it be and
it's very very hard to make
documentation specifically for these
kind of problems especially if it's like
a thing like WordPress where you can
have like so many different plugins and
different WordPress versions and like
different configurations and stuff and
uh I I really enjoy the forum for
exactly these kind of things and I find
it super interesting to browse it as you
say like it's a learning opportunity and
I I think the the other part that I I
find really useful about the help forums
is that a lot of people when they come
there they don't know the terminology
where if they knew which words to search
for they could probably find some help
article about it or some blog post
somewhere uh but oftentimes they they
just don't know like what what is it
actually called like I have this weird
issue that I can try to describe but I
don't know like what what the official
names are and the in in the Forum people
can join in and be like Oh you mean
canonicalization and then they're like
what what kind of weird word is this uh
but at least that gives them a little
bit more of a hook to find out more
about what what they could be doing
differently yeah and there's uh there's
um a big spectrum of people who join in
some of them have little to no
experience some of them are more
familiar with the terms and are just
they have this specific issue that they
might have not been able to find um a
resolution for in the documentation and
it's usually it's usually the the type
of threads that uh people create are
either around more technical stuff um
and or search console related uh issues
and that's usually more related to
crawling
indexing um and there's the other types
of threads which are more related to
ranking like I did everything right and
my site hasn't rank or my traffic went
down and please help me with this uh the
first one the more technical um uh
related issues are easier to solve or
easier for us PES to provide an answer
to because we can always point to the
documentation and figure out in your
specific use case you should do this and
that whereas with the ranking um issues
that's a bit harder because uh it also
tends to generate some heated discussion
from now and then and most of the
time and um what we usually do in those
kind of scenarios is try to go with the
user and try to help them diagnose and
at least eliminate some of the usual
suspects like try to figure out is there
any technical thing that might prevent
you from ranking well um afterwards we
maybe try to look at whatever search
algorithm updates have been
uh launched recently and try to
correlate any traffic changes with those
and afterwards we kind of get to the
part that's more subjective where it's
like content quality and other aspects
such as that where it's more of
everybody's or each B's personal
experience that comes in and tries to
come up with a um with some
recommendations on what the user should
improve I I think ranking questions are
are fundamentally impossible for us as
Google to help with um not not so much
because like we we can't diagnose them
but because there just so many different
ways to kind of look at those problems
and that feels like an optimal situation
where Community input is really valuable
because you could go this way and maybe
create more content on this topic you
could go the other way and like maybe
delete a lot of your junk you could
improve your design you could simplify
the design it's like there there lots of
different op
that can lead to improvements with
ranking and uh sometimes I I feel site
owners just need to see that there are
different ways to do it and no one way
is the right way and that's something
where I I find the experience from the
PE Community is is really helpful um
where it's like if people don't agree
that's I think also a really strong sign
that there just different ways to do it
and that's fine and I think one thing
one one cool thing that we can also do
as pees if is that um if we can figure
things out we can always escalate it
further to the to to our private PE
Community where other PES can pitch in
with their ideas and Community managers
can also choose to escalate it to the
Google product teams so um whenever we
think that okay a user has an issue that
we think shouldn't happen or shouldn't
be there we can try to advocate for them
towards Google in order to either get
the issue fix or at least figure out why
is it happening in the first place so
we're always trying to Advocate on
behalf of the user I'm mentioning this
because some of the users think that we
are just an extension of the Google team
where in fact uh we're trying to help
users based on our own experience and if
we can figure things out we're always
trying to help them by escalating it to
Google and trying to get a response and
maybe along the way solve a bug or
improve some aspect that uh googlers
might have might have not noticed
initially the other thing that you
mentioned is that a lot of it is so you
had like Technical and we had ranking
but a lot of this is also Google search
console and that I find interesting
because I had lots of conversations with
the search console team and I was
visiting them in in the office
um their job is surprisingly hard
because they have to cater to like
professionals who are like managing
hundreds of websites and with like
millions of pages and then there's like
Martin who runs a
bakery and and wants to know how his
website's doing in Search and and
canonicalization what the hell is that
what what does it mean to be discovered
but not crawled and uh and they have to
somehow figure out a way to bring that
into an interface that somewhat to some
degree works for everyone um what what
are like the bigger challenges with with
search console then in the
forums um so some of the bigger
challenges are for people to understand
for example one of the most common
questions I come up against is um how
can I solve these specific notifications
when sometimes there's nothing you need
to do about
oh so for example oh the these pages are
listed as having a canonical tag and not
being indexed and in some cases that's
fine or maybe they're um not indexed du
to them having a no index tag that
WordPress put there because it kind of
makes sense for those pages not to be
indexed but they're shown in search
console and some people uh think that oh
if it's not indexed then it's an issue
that that I need to resolve in order for
my website to perform better true true
yeah and the same with like 404 it's
like oh this it found these 5,000 four
or four pages and like well you've
removed 5,000 products from your shop so
yeah so so would you say the errors in
search console are not
helpful well I think some people don't
really differentiate errors from just
simple
notifications uh and messages that kind
of tell you this is the stat of your
website it's not the case that you
should always do something about it so
it's this is more of a notification
these Pages aren't indexed just so you
know you might have not you might have
de-indexed them intentionally and that's
fine uh there are some cases where some
pages might not be indexed and you might
want to do something about that and
that's where I think a lot of um users
come to the Forum and try to make sure
they understand everything correctly and
try to make sure they're not missing
something and get a bit more information
that they might find in the
documentation okay so kind of they they
would go to The Forum and say like
here's this report in search console
that says I have so many issues what
what do I need to do now that's right
and then you might look at that and say
well it looks like maybe you updated
your website and you removed a lot of
pages so it's expected that all of these
errors are there and it's not that you
have to fix them all but I guess
sometime well I I hope sometimes you
also look at the report and say oh
actually you do have to fix these issues
yep yeah especially when you have things
like craw not index or discover not
indexed a lot of users are worried okay
I did everything right why are these
Pages not getting indexed and us aspes
we can take a look at the website and
figure out uh whether there's anything
that they could actually do it's usually
around those two issues it's not
necessarily something technical that
they need to solve like add this tag or
remove or add this whatever thing uh but
it's um um yeah it's kind of case
specific and we try to figure out
whether there's something they could do
to improve um and get those pages
indexed and sometimes some pages at the
end of the day Google doesn't index all
of the pages so some pages will always
be there and sometimes we just tell
users that as well cool um you you also
built a kind of an add-on to search
console
right um Can can you tell us a little
bit more about the the add-on that you
created sure so um I'm lucky enough to
have a partner in my agency that kind of
focuses on the more creative and
marketing things uh and marketing
aspects of SEO Alexandra and that kind
of allows me more to focus on more
technical stuff and analytics uh and I
think analytics is very important for
both web Masters and seos to try to uh
look at the data and extract insights
and come up with a strategy on how to
improve their website and get more
performance in terms of uh traffic and
Impressions about eight years ago I
found out about
the well search console API I think it
was web Master tools API back then and
um um I'm basically in my SEO work I try
to leverage my technical background
since I finished a Technical University
I have some basic programming skills and
with a lot of tutorials I try to kind of
cut down on the more more repetitive
aspects of SEO kind of the more boring
part and that would leave me more time
to focus on insights coming up with
strategies and so on and as I mentioned
about 8 to 10 years ago there was the
case that within Web Master tools you
had a 90day window uh you now have 16
months but back then you had a 90-day
window where the data was available so
we had to uh remember every month to
download the data and save it uh to be
able to reference it uh later if we
needed to and that was quite manual and
uh we use a lot of Google Sheets and
Google Sheets have has this programming
language that's similar to JavaScript
that allows you to add some
functionality and combined with the API
from Web Master tools uh I built a quick
script to be able to extract data from
from uh Web Master tools and uh extract
it directly into Google Sheets so we can
save it for later use uh afterwards I um
simply buil a UI around it so to make it
easier for both me and my team to to uh
extract the data uh into Google Sheets
and it was just kind of the natural Next
Step was to basically just make it
public and one other thing I noticed
when using the API and that was very
interesting to me is that there seems to
be a lot more data in the API that you
might be able to find in the UI
especially for larger projects and the
data is also structured a lot better to
be able to offer you or make it easier
for you to extract insights for example
you can uh extract all of the queries
and the pages associated with them at
the same time whereas in the UI you kind
of have to click on each query to see
whatever Pages umed clicks or
Impressions uh so you can do that at
scale and sometimes you can download
even extract from the API hundreds of
thousands of rows which makes it very
useful for especially if you have a
larger website that receives a lot of
traffic for a a bunch of different terms
cool nice afterwards um one other
automation I thought about was to make
it even easier to extract those kind of
monthly um uh data from from Web Master
tools and afterwards search console uh
so without even logging into Google
Sheets so the next step was to build an
automation so you can set it up so every
month the add-on extracts that data on
your behalf and puts it into the Google
sheet of your choice and you don't even
have to worry about or remember that oh
I need to log into Google Sheets and
extract the data and that's
nice and after the URL inspections API
was launched I also integrated that in
the into the tool just because that's
kind of a different aspect of scale with
the with the API you can do like
thousands of URL inspections in a few
minutes whereas with the UI you kind of
have to do one by one okay wow it's it
seems like almost like all apis in
search console make it to uh I I I
really love the the kind of monthly
automation I I always thought that was
something that maybe in search console
we we could add something along those
lines as well uh we have I think the
monthly emails that come from search
console insights uh which are little bit
like lightweight over youw but it feels
like I don't know a monthly dump into a
spreadsheet that's that's like a pretty
cool idea like I don't know like why
don't we have
that talk with someone p i I just love
the the search console insid variation
because it's such an an easy to use
broken down look at your website so if
it's like Martin Bakery uh I'm loving it
because I I feel like now we're covering
all through so to speak tiers like the
people who are like I just have a
Business website I just want to know how
it's doing that's like the search
console insights and then like the
average kind of person the average user
or the ad hoc kind of debugging thing
where you just go into search console
and use the UI or the power user like
you clearly are Mii uh who can just like
do pretty much all the things they need
and get all the data they need to then
do their own breakdowns and Analysis in
sheets which I think is pretty cool
right that's one of the main reason I
made the tool public simply because I
figured if I have this issue there's
probably a lot of people in this
community that have this issue as well
as web Masters so yeah I I make I made
it public in order to try to give back
to the SEO
Community um and help other people solve
the same issues that I had cool um have
you tried the big query integration I
have not simply because we're so used
within our team to work with Google
Sheets and I find Google Sheets very
useful to be able to diagnose data and
performance issues and extract insights
without being too much of a technical
user and that's kind of why the add-on
is as simple as possible it's just a way
to here are the parameters give me the
data and it gives you the raw data and
it has privacy in mind as well in the
idea that all of the data is sent
directly to Google Sheets we don't use
any servers everything is is directly
from the Epi into your
spreadsheet um in order to get all of
the raow data that the API has so cool
yeah I I use spreadsheets for everything
it's I I think at home when I say
anything it's like I put it in a
spreadsheet uh if if I need to create
like a giant database for something I
will just put it into a spreadsheet I
know like Martin is probably
this meme of people using spreadsheets
for databases but it's it's especially
Google Sheets is is that thing where
it's almost like a server backend that
you don't really have to like mess with
sure it's it's kind of slow if you kind
of need to do a lot of things but it's
easy to put in it's easy to try things
out copy paste do like simple uh graphs
and things so I I am a big fan of
spreadsheets what do you see in the
future it's like everyone is adding AI
are you adding AI to Google Sheets
that's one feature I I had in mind
although um so I've I still am pretty
involved into the SEO work that our
agency does so I'm I'm getting a lot of
insights based on our own work and
there's some power users that also email
me with feature requests and some of
them are doable in the sense that it is
something that I could Implement in the
future some of them kind of depend on
the apis that Google has available for
example A lot of people are requesting
indexing information which is not yet
available uh in the API but other things
uh right now the tool is just made to
extract the raw data but perhaps you
could or I could Implement something
that would provide some basic insights
similar to the soci console emails that
um every Web Master gets so I think
that's one thing that uh would be not
very hard to implement and would be very
useful to a lot of people afterwards yes
I think with the Advent of llms um a lot
of people would would maybe find it more
useful to use natural language to ask
give me insights about this data what
queries um are performing well what
queries aren't which Pages have issues
and so on and that's actually possible
right now just not with this tool I
actually created a separate tool called
search ches for GPT which is again just
a simple integration of uh chat GPT with
the um uh search console apis to be able
to ask it and retrieve data into into
chat GPT although I don't see that
working for very big projects um maybe
you just want to get like top 10 queries
or top 10 pages and things like that but
probably for big projects you still will
rely on Google Sheets spreadsheets
woohoo spread sheets spread databases I
flinched a little bit as a developer
there yeah oh god um that's that's
really really cool stuff and uh I wonder
is there like a place online where
people can find you and your
tools well I'm I try to be active from
now and then on both Twitter and
Linkedin and uh yeah either my agency
website which is verify. agency or um
the tool website which is search
analytics
for.com and um I guess you can also find
me from now and then in the Google
search Central forums acting as a p
trying to help people up with their
issues o very nice so I know some people
at search console
um if there's anything in the API that
you would like what would that be well
the main thing would be indexing data I
would love to have and a lot of people
would love to have um everything
regarding
indexing uh in the API so basically all
of the indexing reports that you
currently have in search console well
have them directly in the API so you can
extract them in in Google Sheets and
maybe be able to correlate them with the
performance data in order to find out if
you're having issues and try to extract
insights from there and there's also the
issue that within the UI you also have I
think a thousand Pages a thousand rows
available for each indexing report where
whereas in in the API I assume that we
will be able to have a lot more data so
that's again a very useful thing to have
um especially if you have a bigger
website yeah
so I I can't promise anything of course
and like I just hear this now so if
you're listening to this podcast don't
assume that next week search console is
going to launch an indexing report API
but who knows I I think with regards to
the amount of data that might be tricky
because some websites are really large
and like it it will always need to be a
sample of of the actual or of the full
data because I I don't know if if you
have a giant e-commerce site and you
have 50 million 404 Pages like I I don't
see Google being able to give you a list
of 50 million 404 Pages or even as a
site owner it's like what what would you
do with 50 million 404 Pages usually
it's some systemic issue and you fix it
and then it fixes 10 million four or
four pages you don't you don't need to
manually go through and check for like I
don't know millions of pages but okay
indexing API or indexing report API I
guess that's that sounds pretty cool we
we should talk with the search console
people yes definitely would be great
definitely even if it's a lower amount
of data just being able to put it into a
Google Sheets and not having to export
them if if you're doing all of your data
analysis in Google Sheets already having
that uh feature and being able to
extract it easily into Google Sheets
along with all of your performance data
and maybe URL inspections and what have
you would be probably very useful for
people in terms of the data amount I
don't know I I thought the same about
the performance API the search analytics
API and if you have like a million pages
that have receive traffic the API will
actually give you all of the million
pages so it seems that there's a lot
more data that the API is able to
provide versus uh versus the UI which is
always kind of a a sample of the data
yeah yeah cool all right okay we we
should we'll take that back chat with
some people yes yeah we'll take that to
the team awesome well uh thank you so
much for joining us Mii it has been a
huge honor thank you so much for having
me and yeah it's been an honor to be a
guest of the search off the Record
podcast and also thanks John for being
here with me today well thank you Martin
for having me it was a pleasure that I
was allowed to join oh of course you no
of course obviously anyway thanks to all
of you as well for joining us uh and
listening in um into this wonderful
episode and next time on search of the
record we'll be talking about shopping
so thanks again for listening folks and
goodbye goodbye
bye-bye we've been having fun with these
podcast episodes and we hope that you
The Listener have found them both
entertaining and and insightful too feel
free to drop us a note on Twitter @
Google search C or chat with us at one
of the next upcoming events that we go
to if you have any thoughts and of
course don't forget to like And
subscribe thank you so much and goodbye
[Music]