Creating Chatbots in African Languages

0

[ad_1]

The field of natural language processing (NLP) has advanced the furthest in the most widely-used languages like English and Russian. But an emerging body of research is focused on training AI models using African languages.

Thanks to such efforts, the dream of an African language chatbot is edging closer to reality.

Chatbot Research Dominated by English Language

Natural language processing and the large language models that power chatbots like ChatGPT are still relatively new technologies. And to date, research and development has focused on the most spoken languages. 

For example, ChatGPT is available in English, Spanish, French, German, Portuguese, Italian, Dutch, Russian, Arabic, and Chinese. 

The tendency toward language dominance in AI research is largely driven by data availability.

It is estimated that over half of all written content available online is in English. Accordingly, of the datasets needed to train language models, the largest and most readily available are in English, followed by the other most popular languages.

African Languages Pose a Challenge for AI Researchers 

Currently, the world’s largest AI firms are battling it out to build the most advanced chatbots for a handful of languages. But another sphere of research is looking to develop AI tools for less popular languages.

For African languages, the limited availability of training data presents a significant challenge for AI developers.

The linguistic diversity of many African countries further complicates things. For example, South Africa has 11 official spoken languages, and there are thirty-five languages indigenous to the country. With around 2000 languages in use on the continent, amassing vast digital content libraries on an equivalent scale to English would be nearly impossible

Languages spoken in Africa Chatbot Research
Representation of African Linguistic Diversity (Source: ACL Anthology)

Moreover, one recent study identified the lack of basic digital language tools as a factor that inhibits content creation. As the authors observed:

“Creating digital content in African languages is frustrating due to a lack of basic tooling such as dictionaries, spell checkers, and keyboards.”

Nevertheless, efforts are underway to increase the availability of African language data, for instance, by digitizing archival language repositories and making more datasets freely accessible. The work of content creators, curators, and translators is also critical.

Multilingual Models Could Make African Language Chatbots a Reality

Although lacking training data has certainly held African language NLP research back, multilingual pre-trained language models (mPLMs) could help researchers overcome this challenge.

Pre-trained models can be thought of as the building blocks of high-functioning chatbots. However, they still require task-specific fine-tuning in order to deliver conversational outputs.

By acquiring generalizable linguistic information during pretraining, multilingual models are able to interpret the basic structure and outline of related languages without the massive training datasets normally required.

Unsurprisingly, one recent study has shown that language similarity improves model performance. Just like speakers of related languages can often understand each other, models trained with one language can interpret similar languages accurately.

Using this approach, researchers developed an mPLM they called SERENGETI, which covers 517 African languages and language varieties.

This represents a major technological leap forward and a significant improvement on the 31 previously covered African languages.

Disclaimer

In adherence to the Trust Project guidelines, BeInCrypto is committed to unbiased, transparent reporting. This news article aims to provide accurate, timely information. However, readers are advised to verify facts independently and consult with a professional before making any decisions based on this content.

[ad_2]

Source link

Leave A Reply

Your email address will not be published.

bitcoin
Bitcoin (BTC) $ 63,878.00
ethereum
Ethereum (ETH) $ 1,788.75
tether
Tether (USDT) $ 0.999157
bnb
BNB (BNB) $ 588.32
usd-coin
USDC (USDC) $ 0.999599
xrp
XRP (XRP) $ 0.986311
solana
Solana (SOL) $ 74.14
tron
TRON (TRX) $ 0.309619
staked-ether
Lido Staked Ether (STETH) $ 2,265.05
figure-heloc
Figure Heloc (FIGR_HELOC) $ 1.01
hyperliquid
Hyperliquid (HYPE) $ 67.79
zcash
Zcash (ZEC) $ 930.25
monero
Monero (XMR) $ 506.83
dogecoin
Dogecoin (DOGE) $ 0.059215
leo-token
LEO Token (LEO) $ 9.29
usds
USDS (USDS) $ 0.999935
wrapped-steth
Wrapped stETH (WSTETH) $ 2,779.67
whitebit
WhiteBIT Coin (WBT) $ 61.72
chainlink
Chainlink (LINK) $ 9.12
wrapped-bitcoin
Wrapped Bitcoin (WBTC) $ 76,243.00
cardano
Cardano (ADA) $ 0.153846
binance-bridged-usdt-bnb-smart-chain
Binance Bridged USDT (BNB Smart Chain) (BSC-USD) $ 0.998762
stellar
Stellar (XLM) $ 0.140732
wrapped-beacon-eth
Wrapped Beacon ETH (WBETH) $ 2,466.93
ethena-usde
Ethena USDe (USDE) $ 0.999083
dai
Dai (DAI) $ 1.00
near
NEAR Protocol (NEAR) $ 3.33
wrapped-eeth
Wrapped eETH (WEETH) $ 2,465.31
bitcoin-cash
Bitcoin Cash (BCH) $ 208.38
rain
Rain (RAIN) $ 0.004876
quantixai
Quantix Finance (QFI) $ 999.15
susds
sUSDS (SUSDS) $ 1.08
litecoin
Litecoin (LTC) $ 44.24
canton-network
Canton (CC) $ 0.083029
avalanche-2
Avalanche (AVAX) $ 7.10
coinbase-wrapped-btc
Coinbase Wrapped BTC (CBBTC) $ 76,366.00
uniswap
Uniswap (UNI) $ 5.20
tether-gold
Tether Gold (XAUT) $ 3,773.17
blockchain-usd
Blockchain USD (USDB) $ 0.932324
sui
Sui (SUI) $ 0.723048
weth
WETH (WETH) $ 2,268.37
usd1-wlfi
USD1 (USD1) $ 0.998963
the-open-network
Gram (prev. Toncoin) (GRAM) $ 1.02
ripple-usd
Ripple USD (RLUSD) $ 1.00
usdt0
USDT0 (USDT0) $ 0.998824
global-dollar
Global Dollar (USDG) $ 1.00
hedera-hashgraph
Hedera (HBAR) $ 0.057140
paypal-usd
PayPal USD (PYUSD) $ 1.00
ondo-us-dollar-yield
Ondo US Dollar Yield (USDY) $ 1.15
bittensor
Bittensor (TAO) $ 195.07
Shares