last day (17 days later) » 

8:15 AM
2
A: Google Apps Script - How to stream JSON data into BigQuery?

Dimu DesignsThe data payload to BigQuery.Jobs.insert() has to be a blob. You can create that blob from either CSV content or a newline delimited JSON. Newline delimited JSON is a distinct form of JSON that is required by BigQuery. It is not natively supported by Apps Script. However, you should be able to ...

 
Note that Newline delimited json is NOT a json in and of itself, but it's a list of valid json separated by new lines.
 
@Dimu Designs: Thanks. Because the data sent to BigQuery must be blob, so as I understand, the library @google-cloud/bigquery will handle all these tasks that you mentioned for us? You can check the example here: cloud.google.com/bigquery/….
Btw, I checked this article hackernoon.com/json-lines-format-76353b4e588d and I finally know why BigQuery uses ndjson
@Dimu Designs: There are some packages on npm that support ndjson, but for Apps Script, I can not find anything. Can you suggest a function to do the job?
Dimu and @TheMaster: I got another error "GoogleJsonResponseException: API call to bigquery.jobs.insert failed with error: Insufficient Permission". Both the apps script project and the big query are under the same GCP project, also I don't see where to configure the service account for apps script, so I thought it should have the permission automatically, right?
 
@Hoang Setting required scopes might help. See references here
 
@TheMaster: This is my appsscript.json prnt.sc/p90ba3. I had scope "script.external_request"
 
@Hoang Add big query scopes as seen in bigquery.insert documentation.
 
8:19 AM
@TheMaster There is one thing that makes me confused here. My apps script is deployed as addon. I need to store some usage data of addon's users into my BigQuery dataset for marketing, business intelligence purpose. Normally, this is done by using a service account. I see that OAuth scope for bigquery is not suitable here. Because I don't need to access user's bigquery, I only need mine.
@TheMaster Beside it, I added "https://www.googleapis.com/auth/bigquery" into the oauth scopes list, reload the addon to accept the new permission. But I still have that error
 
8:34 AM
@HoangTrinh Try adding https://www.googleapis.com/auth/bigquery.insertdata and make sure the identity of the user accessing the data has appropriate roles in cloud IAM policy.
@HoangTrinh Regarding service account, You'd need elaborate to and fro authentication from apps script to prove that you own the service account and to store credentials of the service account. There are no libraries I'm aware of(except OAuth library) that can help. Another alternative is to publish a web app to execute as "Me".
 
 
2 hours later…
10:19 AM
@TheMaster After checking this tutorial cloud.google.com/docs/authentication/#authentication_strategies, I understand why in cloud functions I didn't need to configure the service account. Basically, GCP automatically do that for us. With Apps Script, somehow we will need to configure it by ourselves . I think I need to use the OAuth library definitely.
 
 
5 hours later…
3:04 PM
@HoangTrinh When it comes to using a service account with Apps Script you're going to need to manage the OAuth flow yourself. Eric Koleda's OAuth2 library should help. You'll need to generate the appropriate credentials for your service account from the GCP console. You'll need to use the generated CLIENT_ID and PRIVATE_KEY along with the requisite scope. Eric Koleda has a sample script up on github illustrating how to use his library in this instance:
https://github.com/gsuitedevs/apps-script-oauth2/blob/master/samples/GoogleServiceAccount.gs
*Correction CLIENT_EMAIL (not CLIENT_ID)
 

  last day (17 days later) »