errors retrieving IUsers list from ID

Mar 8, 2015 at 8:14 PM
Edited Mar 9, 2015 at 9:33 AM
Dear Linvi,
Let me first thank you for your great job here, I hope you could help me in understanding some features I am having problems with.

I am using tweetinvi to download data about a long list of users of whom I have the ID only.
 IEnumerable<long> BloccoIE = UtentiId.ToList() as IEnumerable<long>;
 var BloccoUtenti = Tweetinvi.User.GetUsersFromIds (BloccoIE);
where BloccoIE is a 100 long list of IDs. However, this process works every now and then, and often breaks on var BloccoUtenti (=null) with the following error:
A first chance exception of type 'System.ArgumentNullException' occurred in mscorlib.dll
Additional information: Il valore non può essere null.
If there is a handler for this exception, the program may be safely continued.
I think it might be a Debuff issue (although the cycle should work 180 times before entering in debuff and this is not the case). This is the Debuff I am using:

private static void DebuffNoLog()
    {
        TweetinviEvents.QueryBeforeExecute += (sender, args) =>
        {
            string msgDebuff;

            var queryRateLimit = RateLimit.GetQueryRateLimit(args.Query);
            if (queryRateLimit != null)
            {
                if (queryRateLimit.ResetDateTimeInSeconds > 0)
                {
                    msgDebuff = string.Format("Debuff! restart in {0} seconds at {1}; limit: {2}", queryRateLimit.ResetDateTimeInSeconds, queryRateLimit.ResetDateTime.ToShortTimeString(), queryRateLimit.Limit);
                    Console.WriteLine(msgDebuff);
                }                   
            }

            RateLimit.AwaitForQueryRateLimit(queryRateLimit);
            Console.WriteLine(args.Query);
        };
    }


I hope you will have the time to take a look at it and help me,
in any case, thank you again for your precious work.
Mar 12, 2015 at 12:55 PM
Edited Mar 12, 2015 at 12:55 PM
Hi,

Sorry for the long delay of this reply. I am trying to catch up with all the discussions open and I sometimes left some unanswered.
Concerning your problem it would be very useful to get the information from the ExceptionHandler.

Could you send this information to me please?
IEnumerable<long> BloccoIE = UtentiId.ToList() as IEnumerable<long>;

if (BloccoIE == null)
{
    var lastException = ExceptionHandler.GetLastException().ToString();
}

 var BloccoUtenti = Tweetinvi.User.GetUsersFromIds (BloccoIE);
Regards,
Linvi
Mar 19, 2015 at 10:30 AM
Hi,
I am sorry for not having aswered yo your message any earlier. The mail notification ended up in the spam folder and I stopped looking for an answer.
However, I found out the mistake on my own (I made a mistake in my program and this resulted in a null answer from TW).
I thank you, nonetheless, for your help.


I kindly ask you, if possible, to help me with a problem I have with the getFollowers function. Indeed,
  TwitterCredentials.ExecuteOperationWithCredentials(cred, () =>
                {
                    var followerIds = User.GetFollowerIds(politico);
               }
with this debuff function
    private static void DebuffNoLog()
    {
        TweetinviEvents.QueryBeforeExecute += (sender, args) =>
        {
            string msgDebuff;

            var queryRateLimit = RateLimit.GetQueryRateLimit(args.Query);
            if (queryRateLimit != null)
            {
                if (queryRateLimit.ResetDateTimeInSeconds > 0)
                {
                    msgDebuff = string.Format("Debuff! restart in {0} seconds at {1}; limit: {2}", queryRateLimit.ResetDateTimeInSeconds,  queryRateLimit.ResetDateTime.ToShortTimeString(), queryRateLimit.Limit);
                    Console.WriteLine(msgDebuff);
                }
            }

            RateLimit.AwaitForQueryRateLimit(queryRateLimit);
            Console.WriteLine(args.Query);

        };

gives me a limit of 15 query per 15 min, which slows me down too much. I cannot find the mistake I made but this surely prevents me to go further with my research. Would you please be kind enough to help me with that? (btw the same 15query debuff happens also with the cursored query and does not allow me to retrieve more than 75K followers/user). Should I just wait for 0.9.6 or is there something I am missing in my code?

this is the query I see in the debuff msg:
   api.twitter.com/1.1/followers/ids.json?user_id=158745&count=5000&cursor=-1
best regards,
ML
Mar 19, 2015 at 4:10 PM
Edited Mar 23, 2015 at 11:32 AM
Hi,

I am aware that this limitation is pretty complicated to cope with. Trust me I had to face the same problem.
You do not really have a choice, Twitter Limits the access to the Followers to 15 per 15 minutes (twitter documentation).

When I was doing research we had to use multiple tokens in order to cope with this limitation.
But be aware that I don't think this is in line with Twitter Policy (but there is no other way).

Here is a Work Item that will add this feature but I don't think I will implement it in either 0.9.6.x or 0.9.7.x.

If you want to use multiple tokens here is what you can do :
  • Create a List of IOAuthCredentials
  • Before the query is executed look at your set of credentials and select the one that is going to be reset the soonest
  • If none of your credentials have remaining available operation wait for the soonest available credentials.
  • Otherwise use the soonest refreshed credentials to perform your operations
Please note that the following code will only work with the Source Code version (0.9.6.0) as I have made improvements to the RateLimits.

There are quite few lines of code but they are not complicated to understand.
List<IOAuthCredentials> credentials = new List<IOAuthCredentials>() { /* Add your multiple credentials */};
TweetinviEvents.QueryBeforeExecute += (sender, args) =>
{
    var rateLimits = RateLimit.GetQueryRateLimit(args.QueryURL);
    if (rateLimits == null)
    {
        // RateLimit cannot be identified for the query
        return;
    }

    if (rateLimits.Remaining > 0)
    {
        // We can continue to perform queries with the current credentials
        return;
    }

    if (!credentials.Any())
    {
        // If no credentials have been setup, we have to wait
        RateLimit.AwaitForQueryRateLimit(rateLimits);
        return;
    }

    var credentialsAssociatedWithQueryRateLimit = new Dictionary<ITokenRateLimit, IOAuthCredentials>();
    foreach (var credential in credentials)
    {
        var tokenRateLimits = RateLimit.GetQueryRateLimit(args.QueryURL, credential);
        credentialsAssociatedWithQueryRateLimit.Add(tokenRateLimits, credential);
    }

    var allCredentials = credentialsAssociatedWithQueryRateLimit.Keys.ToArray();
    var credentialsThatCanBeUsed = allCredentials.Where(x => x.Remaining > 0);

    var earliestAvailableRateLimit = GetEarliestAvailableRateLimit(allCredentials);
    var associatedCredentials = credentialsAssociatedWithQueryRateLimit[earliestAvailableRateLimit];

    if (!credentialsThatCanBeUsed.Any()) // There are no credentials with any remaining token for the query
    {
        // Await for the credentials that will be available the soonest
        RateLimit.AwaitForQueryRateLimit(args.QueryURL, associatedCredentials);
        return;
    }

    // We want to use the working credentials for the coming query
    TwitterCredentials.SetCredentials(associatedCredentials);
};
In your case, as you use ExecuteOperationWithCredentials you will have to replace the last line of code so that it updates your 'cred' variable.

Regards,
Linvi
Mar 20, 2015 at 10:31 AM
I thank you for your great help, I will have to dedicate to you my thesis. I really thought (and hoped) it was a 180 query limit.
So, as a matter of fact, my idea to retrieve the followers list of about 100,000 users is not feasible. Do you know if by using the same mobile phone number to create different fake TW accounts I might alert the blacklisters? Any smart hint to have a bunch of tokens and not be caught (e.g. vpn, using old profiles...)? Would it by any chance possible to have some 'special' credential for academic use only?

Again, you're the best. I cannot tell you how much I appreciate your work and your kindness.

Best regards,
ML
Mar 20, 2015 at 11:24 AM
Edited Mar 20, 2015 at 11:29 AM
Hi,

I am not sure how you can get multiple Tokens today without alerting Twitter.
I think the best way would be to have multiple of your friends that give access to your application and use their Tokens to perform the search.

Your study

Let's assume you were not able to get more than 1 token (I guess you will be able to find some friends who can help you ;), and you decided to lower the size of your sample to 36k users to analyze.

Now if you think about your research, most of the users you will be analyzing will have less than 5k followers.
Therefore it implies that most of these users will only require 1 request.

Now Twitter give you the ability to perform 1 request/min with a single Token. As a result you'd be able to analyze 3600 users per day (if they had less than 5k followers).

Now there are of course these famous users who have >5k followers.
What I could suggest to handle these is the following logic.
  • Get the first 5k followers of your user sample (let say 36k users) => 10 days to process with 1 token.
  • After 10 days, you will be able to query your Database and know which users have more than 5k followers (lets call them Famous Users).
  • For the purpose of the example let say your have 1k of Famous Users.
  • Foreach of the Famous Users, you request the next 5k users (once per user) => 1000 minutes => 16 hours.
At this stage you have all the followers of the users who have up to 10k followers (TOTAL : 10 days + 16 hours).

Now your repeat the process and take all the users who have at least 10k followers in your database (lets say 300 maybe).
  • Foreach of them you get the next 5k... => 300 minutes => 5 hours
    TOTAL : 10 days + 21 hours
And we start again with followers who have more than 15k followers...

The End result

By doing this iterative solution, your result will get more and more accurate by the time. The first request will obviously take 10 days (I kind of believe your study will take more than this). And even after this first request you will already be able to distinguish 'famous' users.

The good point of this solution is that even if the process does not complete before your need to write the result of your study, you will have accurate results for more than 99% of your sample. You can eliminate the remaining 1% or just know that your result might be affected by it.

Again, you're the best. I cannot tell you how much I appreciate your work and your kindness.

Thank you very much, I appreciate it. Feel free to let a review on the download page (https://tweetinvi.codeplex.com/releases/view/119313).

Regards,
Linvi
Mar 22, 2015 at 4:48 PM
Dear linvi,
I am trying to implement your lines with the 0.9.6 source code. However, GetEarliestAvailableRateLimit seems not to be found in your source and it is non trivial to me how to write that function.

For the time being, I will select a smaller random subsample in the hope to get some nice result nonetheless. As of the whole project, this might become part of my PhD thesis, so I'll have 3 years to collect data :D

just to be sure: if an account gets blacklisted, does this impact also the non-developer side of the account (I do not want to have all my friends kicked out from TW because of me).
Mar 23, 2015 at 11:15 AM
Hi,

I am trying to implement your lines with the 0.9.6 source code. However, GetEarliestAvailableRateLimit seems not to be found in your source and it is non trivial to me how to write that function.

I have not tested it but it should be something like this:
private ITokenRateLimit GetEarliestAvailableRateLimit(ITokenRateLimit[] tokenRateLimits)
{
    var tokenRateLimitsOrderedByResetDateTime = tokenRateLimits.OrderBy(x => x.ResetDateTimeInMilliseconds).ToArray();
            
    var earliestAvailableRateLimit = tokenRateLimitsOrderedByResetDateTime[0];
    if (earliestAvailableRateLimit.Remaining > 0)
    {
        return earliestAvailableRateLimit;
    }

    for (int i = 0; i < tokenRateLimitsOrderedByResetDateTime.Length; ++i)
    {
        var tokenRateLimit = tokenRateLimitsOrderedByResetDateTime[i];
        if (tokenRateLimit.Remaining > 0)
        {
            return tokenRateLimit;
        }
    }

    return earliestAvailableRateLimit;
}
Also the allCredentials variable should be set to an array :
var allCredentials = credentialsAssociatedWithQueryRateLimit.Keys.ToArray();

just to be sure: if an account gets blacklisted, does this impact also the non-developer side of the account (I do not want to have all my friends kicked out from TW because of me).

I do not have a link with you an official response, when I did my researches the owner of the application was responsible for his actions.
In fact it does not really makes any sense that Twitter would punish users of an application. Any application could use the token to do more than what the user is expected without them knowing.

Punishing users for not being aware of what the application is doing does not seem really possible to me.

Cheers,
Linvi
Apr 13, 2015 at 11:44 PM
dear linvi,
i have finally managed to retrieve the long list of followers I needed. However, i still have problems with the popular guys. I am using your automatic cursored query with only one set of credentials and it gives me back only the first 75K followers (i suppose it is 15x5K limit). I do not find a way to tell to the program to get me the next followers, starting from the 75,000th. Could you help me with that?
This is what i use:
                        var followerQuery = String.Format("https://api.twitter.com/1.1/followers/ids.json?user_id={0}", myPolitician.Id);
                        var results = TwitterAccessor.ExecuteCursorGETCursorQueryResult<IIdsCursorQueryResultDTO>(followerQuery);
                        var followerIds = results.SelectMany(x => x.Ids);
                        IEnumerable<long> ids = followerIds as IList<long> ?? followerIds.ToList();
i have also tried with something like
                        var results = TwitterAccessor.ExecuteCursorGETCursorQueryResult<IIdsCursorQueryResultDTO>(followerQuery, 75000, IdLastFoll);
but this would just give me a null result.

Thank you in advance
ML
Apr 18, 2015 at 11:25 PM
Hi,

Well I am quite surprised to find out that I never talked about the cursors in all this thread.
Your problem right here is that you have not stored the cursor information.

An IIdsCursorQueryResultDTO contains various metadata information to help you execute the next queries.
The most notable property is the NextCursor. This property allows you to start a cursor query at a specific point. This basically allows you to get back and get the next data available when the results are bigger than the default limit (5000 in the context of follower ids).
// Existing Code
var followerQuery = String.Format("https://api.twitter.com/1.1/followers/ids.json?user_id={0}", myPolitician.Id);
var results = TwitterAccessor.ExecuteCursorGETCursorQueryResult<IIdsCursorQueryResultDTO>(followerQuery);

// New Code
var latestResult = results.Last();
var latestCursor = latestResult.NextCursor;

var newResultsStep = TwitterAccessor.ExecuteCursorGETCursorQueryResult<IIdsCursorQueryResultDTO>(followerQuery, cursor: latestCursor);
So what you will need to do is to store this latest NextCursor somewhere so that you can reuse them later on to complete your set of friends.

I am so sorry for not giving you this information before. I must have forgotten. Sorry if this delays your research.

Linvi