Getting raw json from streamed tweets

Nov 23, 2013 at 4:35 PM
Hi All,

I was wondering if there was a way of getting the raw JSON returned from a streamed tweet? I am looking to store the JSON to a database so that I can process this information later?

My intention is to read the raw JSON later during a "processing stage" and during this stage my intention would be to deserialize the raw JSON back to the classes you have provided!


Many Thanks... and great library!


Benjamin
Coordinator
Nov 23, 2013 at 6:53 PM
Hi Benjamin,

There is currently now way to get the json easily from Tweetinvi.
  • The main reason for that is very few developers need the json as is.
  • The second reason is that it would multiply the number of methods by 2 foreach of the classes.
Now as you are requesting it, let me give you some info concerning the processing of Json in Tweetinvi.

Tweetinvi 0.8.x.x
When you perform a request from any object, the request will go through the Token class and the OAuthToken class.
The OAuthToken class retrieves the Json and send it back to the Token class.
The Token class deserialize it and transform it into a Dictionary<string, object>.
All objects (tweet, user, messages...) uses a Token and not the OAuthToken and are consequently unaware of the Json itself.

With this version you will need to call the OAuthToken with the http query and get the json back. You'll also have to construct the query on you own. You can access the queries from the Resources files of Tweetinvi.

Tweetinvi 0.9.0.0 (IMPORTANT)

Tweetinvi 0.9.0.0 is currently in development but the new design will allow me to provide a complete set of methods targeted for developers requesting the Json itself. Tweetinvi 0.9 is not fully implemented and all the projects/namespace/classes name can be changed or deleted.

Even though I understand your need and if you feel comfortable working with an early alpha version of Tweetinvi I will be happy to provide you the objects allowing you to extract the json from Twitter.

To conclude in few words, this is currently complicated but the next release will ease the development of such feature on my side and I will be happy to develop them for you if you are ready to use an Alpha solution.

Kind Regards,
Linvi
Nov 24, 2013 at 12:21 AM
Edited Nov 24, 2013 at 12:22 AM
Hi Linvi,

Thanks for the quick response on this, its a shame that there is currently no way of handling this at the moment although I am happy you are looking into it for future versions.

Since I am looking at this at the moment, I feel I may have a suggestion that may be a little easier to accomplish, I would be happy to handle serialization to JSON myself but at the moment this seems impossible as the Tweet class and associated entities at the moment do not have parameter-less constructors? Could it be a little simpler to provide them with parameterless constructors? and therefore enabling me to use a library such as JSON.NET to serialize them back to JSON myself?

For the moment I am going to simply create carbon copies of the classes and use AutoMapper to copy data from one to the other, that way I can simply serialize and store my own.


Thanks for your help with this, if there is anything I can do to help with this library I would be happy too.

Benjamin
Coordinator
Nov 24, 2013 at 1:23 AM
Hi there,

Could it be a little simpler to provide them with parameterless constructors? and therefore enabling me to use a library such as JSON.NET to serialize them back to JSON myself?

Tweetinvi is an API which grew from getting tweets from a stream into a generic Twitter API. As a consequence the design is not good and it does not allow me to simply create an empty constructor for Json.NET to transform Json into a class.

Please read the next part as it might be of interest to you!

This being said, the new version has a whole new design and uses Json.NET.
In the next days I will work on implementing the ability to retrieve Json from a query. I will provide an update soon and send you an example on how you can access Json with the new design.
Because Tweetinvi 0.9 does not yet contains all the features of the 0.8.3.21, you will be able to only access the JSON of the features existing in the 0.9.0.0 (even if many features have already been implemented).

You can find all the implemented features by downloading the following Excel file.

If a feature has not yet been implemented please ask and I will implement if for you in very brief delays (between 1 hour to 1 day).

Thanks for your help with this, if there is anything I can do to help with this library I would be happy too.

Thanks for the proposal, you can contribute by posting issues, uses cases like you've just done or by being a member of the development team if you wish :)


Hope this gives you a better overview of the current situation of Tweetinvi and where it is going.
Please feel free to ask any question.

Kind Regards,
Linvi
Coordinator
Nov 24, 2013 at 11:42 PM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.
Coordinator
Nov 24, 2013 at 11:58 PM
Hi there,

I have studied a bit more your request and come up with a solution.

Just to give you some news. I have slightly modified the design of 0.9 and it is now able to get json from any query performed on Twitter (not all the queries are implemented but the design is).

Concerning your issue which is more focused on stream. Here are the different steps you can follow to do store the Json from any Stream<Tweet> in tweetinvi (0.8.3.21).
  • Download the source code of Tweetinvi
Update Tweet class
  • Go in Tweetinvi > Tweet.cs
  • Add a field string RawJson { get; set; }
Update SimpleStream
  • Go in Streaminvi > SimpleStream.cs
  • Line 93 after (ITweet t = new Tweet(jsonTweet);)
  • Add : t.RawJson = x;
Update FilteredStream
  • Go in Streaminvi > FilteredStream.cs
  • Line 283 replace : 'return new Tweet(tweetObject);' by
  • 'var t = new Tweet(tweetObject);
    t.RawJson = jsonTweet;'
With these few modification you will now be able to access the RawJson of your Tweet.

I hope this is temporary solution for you until the release of version 0.9.0.0.

Kind Regards,
Linvi
Nov 25, 2013 at 10:11 AM
Thanks Linvi,

I actually came to a very similar solution to this myself but I think yours may be a little more elegant perhaps!
I ended up creating my own RawJsonFilteredStream.cs which derived from the same base class as FilteredStream and essentially called a new delegate which not only accepts ITweet and matching tracks but allowed me to pass a string which represented the original JSON.

This allowed me the most flexibility!

Hopefully if you commit the changes you have made to the library I can use your solution and know that I will not have to maintain my own classes during updates of the library into my solution which would be fantastic :)


So far I have my little data mine service running nicely and it seems to be pretty stable at the moment (been running constantly for around 48 hours without interruption)


Thank you so much for the help!


Benjamin
Coordinator
Nov 25, 2013 at 11:50 AM
Hi Benjamin,

I will not commit this code as I do not think it is relevant for most developers and it will be done in the next version.
So keep working with you RawJsonFilteredStream for now and I will let you know when Tweetinvi 0.9 implement the feature you need.

Kind Regards,
Linvi
Mar 28, 2014 at 4:23 PM
Hi Linvi, hi everybody,

First of all, thanks to you Linvi, and every people that helped in producing this very nice API.

I'm looking for the same thing : get the original json using the filteredstream functionnality.
And even having explored the source code and after many searches both here and over the web, I'm still in the lack of getting any helping clue.
How to get to this goal without modifying many classes and methods in the original solution ?

Could you please help me to get through this? Any advice would be wonderful to me.

Kind regards

Johan
Coordinator
Mar 28, 2014 at 4:43 PM
Hi there,

I think you will be pretty happy to know that I have added this feature in Tweetinvi 0.9.1.0 which is currently under the development lifecycle but should be released this weekend.
var fstream = Stream.CreateFilteredStream();
fstream.AddTrack("hello");
fstream.JsonObjectReceived += (sender, args) =>
{
    Console.WriteLine(args.Value);
};
fstream.StartStreamMatchingAllConditions();
Also please not that args.Value will be changed to args.Json on release.
You can get this version from the Source Code -> Development.

Regards,
Linvi
Mar 29, 2014 at 9:54 AM
Hi there,

You're fast ! This made my day :).

I'll wait for the Nuget update as I can't open the solution with my VS.NET.

Thank you very much.

Best regards,

Johan