NVIDIA TensorRT 7 to speed time-to-market for conversational AI
NVIDIA has introduced inference
software that developers everywhere can use to deliver conversational artificial intelligence (AI) applications, making interactive engagement with AI a reality.
NVIDIA TensorRT 7 — the 7th generation of the company’s inference software development kit — opens the door to smarter human-to-AI interactions, enabling real-time engagement with applications such as voice agents, chatbots and recommendation engines.
It is estimated that there are 3.25 billion digital voice assistants being used in devices around the world, according to Juniper Research. By 2023, that number is expected to reach 8 billion, more than the world’s total population.
TensorRT 7 features a new deep learning compiler designed to automatically optimise and accelerate the increasingly complex recurrent and transformer-based* neural networks needed for AI speech applications. This speeds the components of conversational AI by more than 10x compared to when run on CPUs, driving latency below the 300-millisecond threshold considered necessary for real-time interactions.
“We have entered a new chapter in AI, where machines are capable of understanding human language in real time,” said NVIDIA founder and CEO Jensen Huang at his GTC China keynote in mid-December.
“TensorRT 7 helps make this possible, providing developers everywhere with the tools to build and deploy faster, smarter conversational AI services that allow more natural human-to-AI interaction.”
Some of the world’s largest, most innovative companies are already taking advantage of NVIDIA’s conversational AI acceleration capabilities. Among these is Sogou, which provides search services to WeChat, the world’s most frequently-used application on mobile phones.
“Sogou provides high-quality AI services, such as voice, image, translation, dialogue and Q&A to hundreds of millions of users every day,” said Yang Hongtao, CTO of Sogou.
“By using the NVIDIA TensorRT inference platform, we enable online service responses in real time. These leading AI capabilities have significantly improved our user experience.”
TensorRT 7 speeds up a growing universe of AI models that are being used to make predictions on time-series, sequence-data scenarios that use recurrent loop structures, called RNNs. In addition to being used for conversational AI speech networks, RNNs help with arrival time planning for cars or satellites, prediction of events in electronic medical records, financial asset forecasting and fraud detection.
An explosion of combinations for RNN configurations and functions has created a challenge to rapidly deploy production code that meets real-time performance criteria — causing months-long delays while developers created handwritten code optimisations. As a result, conversational AI has been limited to the few companies with the necessary talent. With TensorRT’s new deep learning compiler, developers everywhere now have the ability to automatically optimise these networks — such as bespoke automatic speech recognition networks, and WaveRNN and Tacotron 2 for text-to-speech — and to deliver the best possible performance and lowest latencies.
The new compiler also optimises transformer-based models like BERT for natural language processing. TensorRT 7 can rapidly optimise, validate and deploy a trained neural network for inference by hyperscale data centres, embedded or automotive GPU platforms. NVIDIA’s inference platform — which includes TensorRT, as well as several NVIDIA CUDA-X AI libraries and NVIDIA GPUs — delivers low-latency, high-throughput inference for applications beyond conversational AI, including image classification, fraud detection, segmentation, object detection and recommendation engines. Its capabilities are widely used by some of the world’s leading enterprise and consumer technology companies, including Alibaba, American Express, Baidu, Pinterest, Snap, Tencent, and Twitter.
Details:
TensorRT 7 will be available in the coming days for development and deployment, without charge to members of the NVIDIA Developer programme from the TensorRT web page. The latest versions of plug-ins, parsers and samples are also available as open source from the TensorRT GitHub repository.
*Transformer' refers to a type of neural network architecture.
NVIDIA TensorRT 7 — the 7th generation of the company’s inference software development kit — opens the door to smarter human-to-AI interactions, enabling real-time engagement with applications such as voice agents, chatbots and recommendation engines.
It is estimated that there are 3.25 billion digital voice assistants being used in devices around the world, according to Juniper Research. By 2023, that number is expected to reach 8 billion, more than the world’s total population.
TensorRT 7 features a new deep learning compiler designed to automatically optimise and accelerate the increasingly complex recurrent and transformer-based* neural networks needed for AI speech applications. This speeds the components of conversational AI by more than 10x compared to when run on CPUs, driving latency below the 300-millisecond threshold considered necessary for real-time interactions.
“We have entered a new chapter in AI, where machines are capable of understanding human language in real time,” said NVIDIA founder and CEO Jensen Huang at his GTC China keynote in mid-December.
“TensorRT 7 helps make this possible, providing developers everywhere with the tools to build and deploy faster, smarter conversational AI services that allow more natural human-to-AI interaction.”
Some of the world’s largest, most innovative companies are already taking advantage of NVIDIA’s conversational AI acceleration capabilities. Among these is Sogou, which provides search services to WeChat, the world’s most frequently-used application on mobile phones.
“Sogou provides high-quality AI services, such as voice, image, translation, dialogue and Q&A to hundreds of millions of users every day,” said Yang Hongtao, CTO of Sogou.
“By using the NVIDIA TensorRT inference platform, we enable online service responses in real time. These leading AI capabilities have significantly improved our user experience.”
TensorRT 7 speeds up a growing universe of AI models that are being used to make predictions on time-series, sequence-data scenarios that use recurrent loop structures, called RNNs. In addition to being used for conversational AI speech networks, RNNs help with arrival time planning for cars or satellites, prediction of events in electronic medical records, financial asset forecasting and fraud detection.
An explosion of combinations for RNN configurations and functions has created a challenge to rapidly deploy production code that meets real-time performance criteria — causing months-long delays while developers created handwritten code optimisations. As a result, conversational AI has been limited to the few companies with the necessary talent. With TensorRT’s new deep learning compiler, developers everywhere now have the ability to automatically optimise these networks — such as bespoke automatic speech recognition networks, and WaveRNN and Tacotron 2 for text-to-speech — and to deliver the best possible performance and lowest latencies.
The new compiler also optimises transformer-based models like BERT for natural language processing. TensorRT 7 can rapidly optimise, validate and deploy a trained neural network for inference by hyperscale data centres, embedded or automotive GPU platforms. NVIDIA’s inference platform — which includes TensorRT, as well as several NVIDIA CUDA-X AI libraries and NVIDIA GPUs — delivers low-latency, high-throughput inference for applications beyond conversational AI, including image classification, fraud detection, segmentation, object detection and recommendation engines. Its capabilities are widely used by some of the world’s leading enterprise and consumer technology companies, including Alibaba, American Express, Baidu, Pinterest, Snap, Tencent, and Twitter.
Details:
TensorRT 7 will be available in the coming days for development and deployment, without charge to members of the NVIDIA Developer programme from the TensorRT web page. The latest versions of plug-ins, parsers and samples are also available as open source from the TensorRT GitHub repository.
*Transformer' refers to a type of neural network architecture.
Comments
Post a Comment