Can this system work for larger documents?

[JoWho]

Hi @philschmid. Thanks for the great tutorial. Essentially I want to use this system to summarize documents, anywhere from 500-1000 words. Do you think Lambda could handle this or would it fail to scale/handle the requests? Any thoughts would be appreciated.

[philschmid]

Hey @JoWho,

How long does the summarization take on a CPU, e.g. Collab?
Which events would you use to trigger the AWS Lambda? If you are going to use HTTP-Events I think it is not 100% suited and an I would rather go with SageMaker -> https://huggingface.co/blog/deploy-hugging-face-models-easily-with-amazon-sagemaker
Best way to find out if it works or not is create a test case.

[JoWho]

@philschmid,

Does HF have a sample summarization API notebook for Sagemaker? Perhaps not exactly my use case, but any general guides or resources would be appreciated.

[philschmid]

You can deploy any summarization model from https://huggingface.co/models?pipeline_tag=summarization without writing code to SageMaker. Every model card has a button at the right top "deploy" where you can select Amazon SageMaker and it generates a code snippet on how to deploy it.

There are two different ways of deploying it either from inside a SageMaker Notebook Instance or SageMaker studio for from you local machine.
Here is a snippet of how to deploy from your local machine. You just need to replace IAM_ROLE_WITH_SAGEMAKER_PERMISSIONS with an IAM role which has the required permissions

from sagemaker.huggingface import HuggingFaceModel
import boto3

iam_client = boto3.client('iam')
role = iam_client.get_role(RoleName='{IAM_ROLE_WITH_SAGEMAKER_PERMISSIONS}')['Role']['Arn']
# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'sshleifer/distilbart-cnn-12-6',
	'HF_TASK':'summarization'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	transformers_version='4.6.1',
	pytorch_version='1.7.1',
	py_version='py36',
	env=hub,
	role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.m5.xlarge' # ec2 instance type
)

predictor.predict({
	'inputs': "The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct."
})