How to get long followers list

May 28, 2015 at 11:45 AM
Hi:

How can I get a long list of user followers for saving in a database?

Cheers!
Coordinator
May 29, 2015 at 9:36 AM
Edited May 29, 2015 at 9:37 AM
Hi,

What do you mean by a long list of user followers? Do you want the ids, or the user objects?
The fact that you store these in a DB has nothing to do with Tweetinvi, does it?

Linvi
May 31, 2015 at 10:05 PM
Edited May 31, 2015 at 10:09 PM
Hi @linvi :

I want the ids, the fact it´s how to make this because the documentation it´s very poor.

Thanks!
May 31, 2015 at 10:09 PM
linvi wrote:
Hi,

What do you mean by a long list of user followers? Do you want the ids, or the user objects?
The fact that you store these in a DB has nothing to do with Tweetinvi, does it?

Linvi
Hi linvi :

I want the ids, the fact it´s how to make this because the documentation it´s very poor.

Thanks!
May 31, 2015 at 10:21 PM
I am using this code:

do
                        {
                            var results = TwitterAccessor.ExecuteCursorGETCursorQueryResult<IIdsCursorQueryResultDTO>(followerQuery, int.MaxValue, cursor);
                            if (results != null)
                            {
                                var resultsArray = results.ToArray();
                                for (var resultIndex = arrayCursor; resultIndex < resultsArray.Length; resultIndex++)
                                {
                                    var resAPI = false;
                                    resAPI = this.setFollowers(resultsArray[resultIndex], scrapId);
                                    if (!resAPI)
                                    {

                                        lists.resumeScrap(DateTime.Now, resultIndex, cursor, scrapId);
                                        lists.setListStatus("pause", scrapId);
                                        return;
                                    }
                                    else {
                                        lists.updateCursor((resultIndex + 1), scrapId);
                                    }
                                }
                                cursor = results.Last().NextCursor;
                            }
                        } while (cursor != 0);
It works the first time, I get 15 arrays with 5000 Ids but it´s and account of 300.000
How to store in db de correct cursor for resuming and continue the scrap in the same position??

Thanks!!
Coordinator
Jun 1, 2015 at 10:04 AM
Edited Jun 1, 2015 at 10:05 AM
Sorry for the documentation. I am currently rewriting a way more detailed documentation. This will be part of the announcement of Tweetinvi 0.9.8.0

It all depends on how much control you want to have over this result. If you don't mind waiting a lot of time before getting ALL the results I would suggest using the RateLimit awaiter.
TweetinviEvents.QueryBeforeExecute += (sender, args) =>
{
    Console.WriteLine(args.QueryURL);
    var rateLimits = RateLimit.GetQueryRateLimit(args.QueryURL);
    if (rateLimits != null)
    {
        Console.WriteLine("{0}/{1}", rateLimits.Remaining, rateLimits.Limit);
    }
};

// Tweetinvi will manage the RateLimits for you and automatically resume the operation when needed.
RateLimit.RateLimitTrackerOption = RateLimitTrackerOptions.TrackAndAwait;
RateLimit.QueryAwaitingForRateLimit += (sender, args) =>
{
    Console.WriteLine("WAITING FOR RATE LIMITS ({0} seconds) : {1}", args.QueryRateLimit.ResetDateTimeInSeconds, args.Query);
};

var ladygaga = User.GetUserFromScreenName("ladygaga");

// This code will take a while as it will wait until all the 300000 users are retrieved
// It would be a good idea to do this in another Thread
var followers = ladygaga.GetFollowerIds(300000).ToArray();
If you want to do by iteration, you will need to store the NextCursor value of the latest set of data.
var results = TwitterAccessor.ExecuteCursorGETCursorQueryResult<IIdsCursorQueryResultDTO>("", int.MaxValue, -1);
var nextCursorToUse = results.Last().NextCursor;
I would also advise that you take a look into the RateLimit class to easily await for new operation tokens to be available to perform your queries.

Cheers,
Linvi
Jun 1, 2015 at 10:47 AM
Thanks for your reply!

I dónt need the rate limit method beacause my software it´s not waiting every time beacause it´s a service that starts and stop and I have to resumen the search operation.

I need to check in a running service the iteration but the cursor code snippet it´s not working.

The query is:

var results = TwitterAccessor.ExecuteCursorGETCursorQueryResult<IIdsCursorQueryResultDTO>("https://api.twitter.com/1.1/followers/ids.json?screen_name=@shakira", int.MaxValue, 1502568644149522561);

The next cursor value is ->1502568644149522561

The first time with cursor -1 I have got and array with 15 elements of 5000 ids but results.Last().NextCursor returns 1502568644149522561 cursor.

I think the error it´s the cursor value but I am doing how you say.

What´s the problem??

Thanks you so much for your support!
Regards
Coordinator
Jun 1, 2015 at 12:24 PM
Edited Jun 1, 2015 at 12:25 PM
Here is how I would do it (and I checked it works as expected).
RateLimit.AwaitForQueryRateLimit("https://api.twitter.com/1.1/followers/ids.json?screen_name=shakira");
var results = TwitterAccessor.ExecuteCursorGETCursorQueryResult<IIdsCursorQueryResultDTO>("https://api.twitter.com/1.1/followers/ids.json?screen_name=shakira").ToArray();
if (results.Any())
{
    var nextCursorToUse = results.Last().NextCursor;

    var rateLimits = RateLimit.GetQueryRateLimit("https://api.twitter.com/1.1/followers/ids.json?screen_name=shakira");

    Console.WriteLine("Restarting queries at {0}", rateLimits.ResetDateTime.ToLongTimeString());
    RateLimit.AwaitForQueryRateLimit(rateLimits);

    var results2 = TwitterAccessor.ExecuteCursorGETCursorQueryResult<IIdsCursorQueryResultDTO>("https://api.twitter.com/1.1/followers/ids.json?screen_name=shakira", cursor: nextCursorToUse);
}
Linvi
Coordinator
Jun 1, 2015 at 12:57 PM
PS : I will improve the answer when I have some more time.
Coordinator
Jun 2, 2015 at 10:28 AM
Edited Jun 2, 2015 at 11:53 AM
As promised, please find an improved version of the code:
// Retrieve all the follower ids the RateLimit tokens authorize in one go
private IEnumerable<IIdsCursorQueryResultDTO> GetFollowerIds(string username, long cursor, out long nextCursor)
{
    var query = string.Format("https://api.twitter.com/1.1/followers/ids.json?screen_name={0}", username);

    // Ensure that we can get some information
    RateLimit.AwaitForQueryRateLimit(query);
    var results = TwitterAccessor.ExecuteCursorGETCursorQueryResult<IIdsCursorQueryResultDTO>(query, cursor : cursor).ToArray();

    if (!results.Any())
    {
        // Something went wrong. The RateLimits operation tokens got used before we performed our query
        RateLimit.ClearRateLimitCache();
        RateLimit.AwaitForQueryRateLimit(query);
        results = TwitterAccessor.ExecuteCursorGETCursorQueryResult<IIdsCursorQueryResultDTO>(query, cursor : cursor).ToArray();
    }

    if (results.Any())
    {
        nextCursor = results.Last().NextCursor;
    }
    else
    {
        nextCursor = -1;
    }

    return results;
}

// Main method iterating to get all the available follower ids
static void Main()
{
    RateLimit.RateLimitTrackerOption = RateLimitTrackerOptions.TrackOnly;

    var followerIds = new List<long>();
    long nextCursor = -1;

    do
    {
        var firstBatchOfFollowerIds = GetFollowerIds("shakira", nextCursor, out nextCursor);
        followerIds.AddRange(firstBatchOfFollowerIds.SelectMany(x => x.Ids));
    } 
    while (nextCursor != -1 && nextCursor != 0);
}
Marked as answer by linvi on 7/7/2015 at 3:12 AM
Jun 2, 2015 at 3:18 PM
Linvi you are the machine!, I have implemented the code with the example before last and It works like a charm.

I need the same code for searching user by specific keywords.

Thank you so much!!!
Jun 18, 2015 at 6:35 PM
Hi Linvi:

I was going crazy but when I am scraping a long list of users the program fails with no exception.

It´s very strange and I didnt´found the way for fix it.

Can you help me?

Thanks
King Regards
Coordinator
Jun 18, 2015 at 7:33 PM
I am sorry but I will not be able to help you if I have no exception.
Maybe you do not catch the Exception.

Try this in Visual Studio:

DEBUG > EXCEPTIONS > Check the 'Common Language Runtime Exceptions' thrown column.

Cheers,
Linvi
Jun 18, 2015 at 8:14 PM
Thanks Linvi, I will test it!!

King Regards
Jun 19, 2015 at 2:10 PM
No exception throwed, O_O

The results console shows -> 0x4330 thread ended with code 259 ( 0x103 ) .

Any idea??

Thanks!!